Cross-Architecture Steering Transfer in Language Models: A Systematic Empirical Study

# Cross-Architecture Steering Transfer in Language Models: A Systematic Empirical Study
Source: [https://arxiv.org/html/2608.05164](https://arxiv.org/html/2608.05164)
###### Abstract

Independently trained large language models may develop shared internal representations of semantic concepts despite architectural differences—but whether this geometric similarity has functional consequences for cross\-model behavioural control has not been systematically tested\. In this work, we present the first systematic evaluation of whether the shared geometry of independently trained LLMs is*functionally exploitable*for cross\-model steering transfer, and show that it is, conditionally: concept directions extracted from one model can steer a different, independently trained model when sufficient representational capacity has been reached\. We study five open\-weight models spanning three parameter scales \(0\.8B–8B\) and two architectural lineages, training one Sparse Autoencoder per model across 15 semantic domains and testing alignment across all 20 directed model pairs\. Specifically, we observe a suggestive discontinuity near 1\.7B parameters: at≥\\geq1\.7B scale, 47–49% of cross\-model feature pairs validate \(Pearsonr≥0\.60r\\geq 0\.60, Procrustes cosines 0\.895–0\.956\), while alignment degrades sharply below 0\.8B despite raw activation geometry being preserved \(withn=1n\{=\}1model per tier below 7B this cannot yet be confirmed as a systematic scaling law\)\. Leveraging this shared structure, cross\-model steering vectors \(B3\-TI\) achieve a 71\.0% win rate across 15 supervised concepts and five models versus 68\.0% for same\-model native vectors; a single universal vector from an unsupervised shared concept space achieves 67\.3% in 4 of 5 models without any per\-model supervision\. Transfer degrades for models below 1\.7B scale and for one model with generation instability, confirming that functional exploitability of the shared geometry requires sufficient representational capacity\. Our findings underscore the importance of scale thresholds in mechanistic interpretability: tools validated at 7B scale may not transfer to smaller models or frontier architectures without revalidation\. More broadly, we provide the first functional complement to the Platonic Representation Hypothesis—showing that geometric convergence across independently trained LLMs is strong enough to support cross\-model behavioural control without fine\-tuning, under the identified scale conditions\.

## 1Introduction

A central question in mechanistic interpretability is whether the internal representations of independently trained language models are*universal*: not merely similar in the features they respond to, but geometrically aligned in a way that supports functional operations such as steering, suppression, and cross\-model transfer\. If such shared geometry is not just a structural curiosity but is actually*exploitable*—if a concept direction from one model can causally control a different model it has never been trained on—that would mark a fundamental shift in how we think about cross\-architecture interpretability tools\.

The Platonic Representation Hypothesis\(Huhet al\.,[2024](https://arxiv.org/html/2608.05164#bib.bib13)\)provides geometric evidence that representations across modalities and architectures converge toward a shared statistical model of reality\. Whether this shared geometry has*functional*consequences—specifically, whether a concept direction extracted from one model can causally steer a different, independently trained model—remains an open question\. Prior steering\-vector work\(Turneret al\.,[2023](https://arxiv.org/html/2608.05164#bib.bib10); Zouet al\.,[2023](https://arxiv.org/html/2608.05164#bib.bib12)\)demonstrates functional control*within*a single architecture but does not test cross\-architecture transfer\.

This work addresses that question in three stages:\(1\) we establish that cross\-architecture feature\-level universality existsfor models at≥\\geq1\.7B parameters, using SAE\-extracted features aligned across all 20 directed pairs of five independently trained models;\(2\) we explore whether the shared structure is functionally usable for steering, measuring whether concept directions injected at inference time causally shift output distributions in the expected direction;\(3\) we demonstrate that cross\-model steering transfer works in practice, with cross\-model vectors outperforming naive baselines and a single unsupervised universal vector generalising to 4 of 5 models without per\-model supervision\.

Our study makes the following contributions:

1. 1\.Evidence for cross\-architecture feature\-level universality\.Among four models at≥\\geq1\.7B scale, 47–49% of MNN\-extracted feature pairs validate cross\-model \(Pearsonr≥0\.60r\\geq 0\.60, Procrustes cosines 0\.895–0\.956, co\-activation enrichment28\.6×28\.6\\timesabove permutation null\), with a suggestive discontinuity near 1\.7B below which alignment degrades substantially \(n=1n\{=\}1model per tier below 7B; systematic scaling requires a broader model sweep\)\.
2. 2\.Exploration of functional steering transferability\.We evaluate whether the shared geometry supports causal concept control across architectures, using a bidirectionality criterion \(rbidir=\|Δ−\|/\|Δ\+\|r\_\{\\text\{bidir\}\}=\|\\Delta^\{\-\}\|/\|\\Delta^\{\+\}\|\) as evidence that injected directions genuinely encode the target concept rather than one\-directional saturating triggers\. Medianrbidir=1\.71r\_\{\\text\{bidir\}\}=1\.71for B3\-TI vectors \(82% of model–concept pairs withrbidir≥0\.5r\_\{\\text\{bidir\}\}\\geq 0\.5\) confirms T2 causal evidence\.
3. 3\.Demonstration that cross\-model transfer works\.B3\-TI cross\-model vectors achieve a 71\.0% win rate across 15 supervised concepts and five models, compared to 68\.0% for same\-model native vectors and 65\.7% for a naive zero\-projection baseline \(the 3\.0pp gap is directional; pairedt​\(4\)=1\.85t\(4\)\{=\}1\.85,p=0\.14p\{=\}0\.14, underpowered atn=5n\{=\}5models\)\. A single universal vector from an unsupervised shared concept space achieves 67\.3% across 11 unsupervised concepts in 4 of 5 models without per\-model supervision\.
4. 4\.Reproducible pipeline and fully released artefacts\.All intermediate and final outputs are released on Hugging Face: five per\-model residual\-stream activation datasets \(A2\), five trained TopK SAEs \(A3\), 4,100\+ labelled feature files \(A4\), native SAE\-decoder and CAA steering vectors per concept \(A5\), 20 trained cross\-model MLP bridges \(B1\), full pairwise alignment results and statistics \(B2\), B3\-TI cross\-model vectors for all 20 directed pairs \(B3\), the trained universal Global MLP \(C1\), universal concept cluster mappings and results \(C2\), and 55 C3\-Dec / C3\-EncDec universal steering vectors \(C3\), together with all evaluation results \(D1\) and the complete steering evaluation dataset\.

The paper is organised as follows\. Section[2](https://arxiv.org/html/2608.05164#S2)reviews related work\. Section[3](https://arxiv.org/html/2608.05164#S3)describes the end\-to\-end pipeline\. Section[4](https://arxiv.org/html/2608.05164#S4)presents results: geometric universality \(T1, §[4\.1](https://arxiv.org/html/2608.05164#S4.SS1)\), domain structure \(§[4\.2](https://arxiv.org/html/2608.05164#S4.SS2)\), and functional transfer \(T2, §[4\.3](https://arxiv.org/html/2608.05164#S4.SS3)\)\. Section[5](https://arxiv.org/html/2608.05164#S5)discusses scope, failure modes, and future directions\.

## 2Related Work

Brickenet al\.\([2023](https://arxiv.org/html/2608.05164#bib.bib5)\)showed SAEs decompose residual\-stream activations into near\-monosemantic features;Gaoet al\.\([2024](https://arxiv.org/html/2608.05164#bib.bib8)\)established TopK SAEs as the scaling\-efficient variant;Cunninghamet al\.\([2023](https://arxiv.org/html/2608.05164#bib.bib9)\)evaluated interpretability across GPT\-2 scales—we build on this by training TopK SAEs on five architectures and using their features as the cross\-model alignment substrate\.Turneret al\.\([2023](https://arxiv.org/html/2608.05164#bib.bib10)\)introduced Contrastive Activation Addition \(CAA\) andZouet al\.\([2023](https://arxiv.org/html/2608.05164#bib.bib12)\)demonstrated linear representation probes for safety concepts, both operating*within*a single model; we evaluate whether such directions transfer*across*architectures\.Kornblithet al\.\([2019](https://arxiv.org/html/2608.05164#bib.bib15)\)andNguyenet al\.\([2021](https://arxiv.org/html/2608.05164#bib.bib16)\)established CKA and Procrustes alignment as representational similarity tools;Raghuet al\.\([2017](https://arxiv.org/html/2608.05164#bib.bib14)\)contributed SVCCA;Huhet al\.\([2024](https://arxiv.org/html/2608.05164#bib.bib13)\)synthesised these into the Platonic Representation Hypothesis \(T1 convergence\)—we provide the first T2 functional complement showing T1 alignment is strong enough for zero\-shot steering transfer\.Haghverdiet al\.\([2018](https://arxiv.org/html/2608.05164#bib.bib17)\)introduced MNN for cross\-dataset cell matching in genomics, adapted here for cross\-architecture SAE feature alignment;Conneauet al\.\([2018](https://arxiv.org/html/2608.05164#bib.bib18)\)applied MNN to cross\-lingual embeddings\.Wanget al\.\([2024](https://arxiv.org/html/2608.05164#bib.bib1)\)andLanet al\.\([2024](https://arxiv.org/html/2608.05164#bib.bib3)\)find high structural SAE\-feature similarity across LLM pairs under rotation\-invariant measures, establishing geometric universality but not testing causal steering transfer;Gurneeet al\.\([2024](https://arxiv.org/html/2608.05164#bib.bib4)\)find 1–5% of neurons are universal across GPT\-2 seeds at neuron granularity, whereas we operate at SAE feature granularity across architecturally distinct models\.Lindseyet al\.\([2024](https://arxiv.org/html/2608.05164#bib.bib2)\)introduce sparse*crosscoders*that jointly encode activations from multiple models to produce a shared feature set for model diffing—our Global MLP \(C1\) is conceptually related but optimises explicitly for cross\-model steering transfer via contrastive alignment rather than feature decomposition\.

## 3Method

The pipeline spans eleven steps across three tracks \(see Figure[3](https://arxiv.org/html/2608.05164#A1.F3)in Appendix[A](https://arxiv.org/html/2608.05164#A1)for a visual overview\):

Track A — Single\-model*\(run per architecture\):* A1Corpus: 394,508\-passage corpus across 15 semantic domains\. A2Activations: residual\-stream mean\-pool at≈50%\{\\approx\}50\\%depth \(released\)\. A3SAE training: one TopK SAE per model, 16,384 latents \(released\)\. A4Feature labelling: delta\-selection \+ LLM\-judge scoring \(4,100\+ files released\)\. A5Native vectors: SAE\-decoder and CAA vectors per concept \(released\)\.

Track B — Pairwise cross\-model*\(20 directed pairs\):* B1Feature alignment \+ MLP bridge training via MNN matching \(released\)\. B2Pair validation: Pearsonr≥0\.60r\\\!\\geq\\\!0\.60, Procrustes cosine, enrichment \(released\)\. B3B3\-TI cross\-model steering vectors via B1 bridge \(released\)\.

Track C — All\-model universal*\(all 5 models jointly\):* C1Global MLP: joint encoder/decoder over all five activation spaces \(released\)\. C2Universal concept discovery: HDBSCAN yields 11 canonical concepts \(released\)\. C3Universal vectors: C3\-Dec and C3\-EncDec, no per\-concept supervision \(released\)\.

Track D — Functional evaluation: D1Win rate, bidirectionality, and concept\-scoreΔ\\Deltaacross 30 prompts×\\times15 concepts×\\times9 strengths \(all results released\)\.

Full derivations, hyperparameters, and failure analyses are in the appendix; each paragraph below points to the relevant section\.

#### Models \(Table[1](https://arxiv.org/html/2608.05164#S3.T1)\)\.

We study five open\-weight decoder\-only models spanning three parameter scales \(GPT\-2\-large 0\.8B, Gemma\-2\-2B 1\.7B, three 7B models: LLaMA\-3\.1\-8B, Mistral\-7B, DeepSeek\-7B\) and two architectural lineages \(GPT\-2 absolute\-positional MHA; LLaMA\-family RoPE variants including standard MHA, GQA, and sliding\-window GQA\)\. LLaMA \(Hermes\-3, SFT only\) is the sole instruction\-tuned model; the remaining four are unmodified base checkpoints, included to test whether fine\-tuning disrupts representational universality \(Section[4\.1](https://arxiv.org/html/2608.05164#S4.SS1)\); it does not\.

Table 1:Models used in this study\.
#### A1–A2: Corpus and activations\.

We assemble 394,508 passages from 17 HuggingFace datasets covering 15 semantic domains: Python code \(3 granularity levels\), SQL, four mathematics subsets \(GSM8K, MetaMath, MATH\-plus, NuminaMath\-CoT\), creative writing, academic abstracts, biomedical, legal, news, open\-domain QA, and two general prose sources\. For each passage we record the mean\-pooled residual\-stream hidden state at the target layer \(≈\\approx50% depth\), then z\-score normalise per dimension to remove scale disparities of up to12×12\\timesacross architectures\. Full specifications are in Appendix[B](https://arxiv.org/html/2608.05164#A2)\.

#### A3: Sparse Autoencoder training\.

One TopK SAE\(Gaoet al\.,[2024](https://arxiv.org/html/2608.05164#bib.bib8)\)is trained per model:64×64\\timesexpansion for GPT\-2\-large and Gemma \(81,920 and 147,456 features respectively\),128×128\\timesfor the three 7B models \(524,288 features each\)\. Ghost gradients are used for 7B models to prevent dead\-neuron accumulation\. All five SAEs converge to 0% dead features; reconstruction losses range from 0\.065 \(GPT\-2\-large\) to 0\.250 \(LLaMA\)\. Per\-model hyperparameters and losses are in Appendix[C](https://arxiv.org/html/2608.05164#A3)\.

#### A4: Feature selection and domain assignment\.

For each of the 15 supervised domains, the top\-150 SAE features are selected by differential deltaδc​\[f\]=F¯pos,c​\[f\]−F¯neg,c​\[f\]\\delta\_\{c\}\[f\]=\\bar\{F\}\_\{\\text\{pos\},c\}\[f\]\-\\bar\{F\}\_\{\\text\{neg\},c\}\[f\]; domain assignment isarg⁡maxc⁡\|δc​\[f\]\|\\arg\\max\_\{c\}\|\\delta\_\{c\}\[f\]\|\. This yields 666–883 labelled features per model \(4,100 total\); full configuration in Appendix[D](https://arxiv.org/html/2608.05164#A4)\.

#### A5: Native steering vectors\.

Two vector types are computed per \(model, concept\)\.*SAE\-decoder vectors*: a confidence\-weighted sum of the top\-3 SAE decoder columns for the concept’s highest\-δc\\delta\_\{c\}features,ℓ2\\ell\_\{2\}\-normalised\.*Contrastive Activation Addition \(CAA\) vectors*\(Turneret al\.,[2023](https://arxiv.org/html/2608.05164#bib.bib10)\): mean residual\-stream difference between 50 positive and 50 negative passages for the concept,ℓ2\\ell\_\{2\}\-normalised\. The cosine similarity between the two types serves as a within\-model quality check before cross\-model transfer is attempted\.

#### B1: Cross\-model feature alignment\.

For each of the 20 directed model pairs, MNN\-selected feature pairs with composite alignment score≥0\.70\\geq 0\.70are retained \(3,308 validated pairs total\); a 67M\-parameter MLP bridge is trained per directed pair for B3\-TI\. Per\-pair statistics are in Appendix[E](https://arxiv.org/html/2608.05164#A5)\.

#### B2: Alignment validation\.

Each B1 pair is validated using Pearsonr≥0\.60r\\geq 0\.60, permutation enrichment \(BH\-FDR\), Cohen’sdd, Spearmanρ\\rho, CCC, and RSA\. SAE\-free Procrustes cosines on raw activations provide architecture\-independent upper bounds\. Full statistics are in Appendix[F](https://arxiv.org/html/2608.05164#A6)\.

#### B3: Cross\-model steering vectors\.

*B3\-TI \(Translation Injection\)*: the guide’s A5 SAE\-decoder vector is encoded to sparse features, projected through the B1 MLP bridge, and decoded through the target SAE decoder \(ℓ2\\ell\_\{2\}\-normalised\)\. A*naive baseline*\(dimension\-matched,ℓ2\\ell\_\{2\}\-normalised\) provides a zero\-learning\-cost reference\. Two discarded B3 variants are described in Appendix[O](https://arxiv.org/html/2608.05164#A15)\.

#### C1: Global MLP\.

Five per\-model encoders \(dmodel→2048→512d\_\{\\text\{model\}\}\\to 2048\\to 512\-d shared space\) with symmetric decoders are trained jointly using MSE reconstruction loss plus NT\-Xent contrastive loss\(Chenet al\.,[2020](https://arxiv.org/html/2608.05164#bib.bib19)\)\(τ=0\.1\\tau\{=\}0\.1\), aligning same\-passage representations from different models while preserving per\-model structure\. Training runs for 200 epochs on 8×\\timesA100\-80GB; final reconstruction loss is 0\.013 with 0% dead concept\-space neurons\. Architecture and training details are in Appendix[G](https://arxiv.org/html/2608.05164#A7)\.

#### C2: Universal concept discovery\.

The 4,100 A4\-labelled SAE feature vectors are projected into C1 concept space, dimensionality\-reduced with UMAP \(512→30512\\to 30\-d, reducing noise from 58% to 15\.8%\), and clustered with HDBSCAN, yielding 113 clusters spanning≥\\geq2 model architectures\. Each cluster is LLM\-labelled \(Claude, temperature 0\) and human\-reviewed; after deduplication and noise removal,11 canonical universal conceptswith 5/5 model coverage emerge\. Full cluster tables and canonical mapping are in Appendix[H](https://arxiv.org/html/2608.05164#A8)\.

#### C3: Universal steering vectors\.

*C3\-Dec*: the C2 cluster centroid is decoded through C1’s per\-model decoder and SAE decoder to native hidden\-space; polarity resolved by sign\-agreement with A5 CAA vectors\.*C3\-EncDec*: the guide’s A5 vector is encoded through C1’s guide encoder, decoded through the target’s C1 decoder and SAE, and averaged over all guide models \(polarity correct by construction\)\. Both require no supervision, yielding11×5=5511\{\\times\}5=55vectors per type atO​\(n\)O\(n\)extension cost versusO​\(n2\)O\(n^\{2\}\)for B3\-TI\.

#### D1: Functional evaluation\.

All six vector types are evaluated under a shared protocol: 30 prompts, nine\-point strength sweep \(s∈\{−5,−3,−2,−1,0,1,2,3,5\}s\\in\\\{\-5,\-3,\-2,\-1,0,1,2,3,5\\\}\), additive residual\-stream injection, and DeBERTa\-v3\-large NLI concept scoring\(Laureret al\.,[2023](https://arxiv.org/html/2608.05164#bib.bib26)\)\. Primary metrics: win rate \(Δ\>0\\Delta\>0fraction\) and bidirectionalityrbidir=\|Δ−\|/\|Δ\+\|r\_\{\\text\{bidir\}\}=\|\\Delta^\{\-\}\|/\|\\Delta^\{\+\}\|; outputs with repetition\>\>0\.40 excluded\.deepseekats=1s=1only;gpt2\-largeats≤3s\\leq 3\. Full protocol in Appendix[K](https://arxiv.org/html/2608.05164#A11)\.

## 4Results

### 4\.1Geometric Universality \(T1\)

Table[2](https://arxiv.org/html/2608.05164#S4.T2)reveals a suggestive discontinuity at the 0\.8B/1\.7B boundary:gemma\-2\-2b\(1\.7B\) achieves a 46\.9% B2 validation pass rate, statistically indistinguishable from same\-scale 7B pairs \(48\.9%,Δ=2\\Delta\{=\}2pp\), whilegpt2\-large\(0\.8B\) reaches only 29\.9–33\.3%\. Withn=1n\{=\}1model at each tier below 7B this observation cannot yet be confirmed as a quantitative scaling law, but it is the sharpest empirical signal of the T1 track and motivates the 1\.7B threshold used throughout\. SAE\-free Procrustes cosines \(Table[3](https://arxiv.org/html/2608.05164#S4.T3)\) confirm that the geometric substrate is not an artefact of the SAE decomposition: all ten undirected pairs exceed 0\.84, with 7B↔\\leftrightarrow7B pairs reaching 0\.895–0\.956\. Across all 3,308 validated pairs, the mean Pearsonrris 0\.512 \(median 0\.545\) and co\-activation enrichment is28\.6×28\.6\\timesabove the permutation null \(range 5–1,259×\\times\)\. Thellamainstruction\-tuned model’s pairwise pass rates \(42\.6–47\.7%\) are within 2pp of same\-scale base models, consistent with SFT fine\-tuning not disrupting representational alignment, though a matched base–IT ablation within the same model family \(e\.g\., Llama\-3\.1\-8B base vs\. Hermes\-3\-Llama\-3\.1\-8B\) would be needed to confirm this definitively\. Full pairwise statistics are in Appendix[F](https://arxiv.org/html/2608.05164#A6)\.

Table 2:Alignment validation pass rate \(%\) by scale tier\. The 45\.5% overall rate is a mixture statistic; scale\-tier rows are the scientifically correct reporting unit\.Table[3](https://arxiv.org/html/2608.05164#S4.T3)reports SAE\-free Procrustes cosines for each of the ten undirected pairs, computed by aligning raw activations via CCA \(64 components\) followed by Procrustes rotation and then measuring cosine similarity between the resulting concept direction vectors\. These constitute optimised upper bounds on raw\-space geometric alignment, independent of any SAE decomposition, and we report them explicitly as such rather than as unoptimised cosines\.

Table 3:SAE\-free Procrustes cosines \(CCA \+ Procrustes optimised upper bound; no SAE involvement\)\. All pairs exceed 0\.84\.
### 4\.2Domain Structure

Table[4](https://arxiv.org/html/2608.05164#S4.T4)reports B2 domain\-level statistics\. Six domains achieve≥\\geq50% pass rate with Cohen’sd\>0\.4d\>0\.4at≥\\geq1\.7B scale \(math\_olympiad96\.5%,code\_instructions86\.7%,academic\_writing76\.5%,math\_gsm8k60\.0%,math\_competition55\.9%,news\_reporting50\.9%\)\. Domains with highly structured, syntactically distinct registers \(formal code, mathematical proof style, academic writing\) generalise strongly; domains that overlap with general web text \(QA, biomedical\) perform poorly\. Domains with negative Cohen’sddindicate that the labelled SAE features are not discriminative for those domains in the MNN\-matched subset; we report them in full for reproducibility\. Full domain breakdown is in Appendix[F](https://arxiv.org/html/2608.05164#A6)\.

Table 4:B2 domain\-level validation results \(3,308 pairs, all 5 models\)\. Bold rows are included in the main universality claim; remaining rows are reported in full for reproducibility\. Negative Cohen’sddindicates feature activations are not discriminative for that domain\.
### 4\.3Functional Transfer \(T2\)

#### Causal evidence \(bidirectionality\)\.

For the four models that sustain valid outputs at\|s\|≥2\|s\|\\geq 2,rbidir=\|Δ−\|/\|Δ\+\|r\_\{\\text\{bidir\}\}=\|\\Delta^\{\-\}\|/\|\\Delta^\{\+\}\|has a median of 1\.71 for B3\-TI vectors \(82% of model–concept pairs withrbidir≥0\.5r\_\{\\text\{bidir\}\}\\geq 0\.5\), confirming that sign\-flipping consistently suppresses as well as amplifies the target concept\. This rules out one\-directional saturating triggers and provides a necessary consistency condition for causal concept control\. We distinguish this from mechanistic causal tracing \(T3, deferred as future work, Section[5](https://arxiv.org/html/2608.05164#S5)\): followingTurneret al\.\([2023](https://arxiv.org/html/2608.05164#bib.bib10)\)andZouet al\.\([2023](https://arxiv.org/html/2608.05164#bib.bib12)\)we use bidirectionality as a functional prerequisite for genuine concept encoding rather than a mechanistic proof\.

#### Transfer efficiency \(win rates\)\.

Table[5](https://arxiv.org/html/2608.05164#S4.T5)reports win rates across 15 supervised concepts and 30 evaluation prompts\. B3\-TI achieves 71\.0% versus 68\.0% for native SAE\-decoder and 65\.7% for the naive zero\-projection baseline, confirming that the MLP bridge provides systematic benefit over the zero\-learning\-cost alternative\.deepseek\-llm\-7b’s low B3\-TI win rate \(21\.7%\) reflects repetition collapse ats≥2s\\geq 2\(Appendix[I](https://arxiv.org/html/2608.05164#A9)\); restricted tos=1s=1the rate is 30\.0%, consistent with other models\. For the unsupervised track, C3\-Dec achieves 67\.3% win rate across 11 concepts with 4 of 5 models above 60%, including 100% formistral\-7b\(Table[6](https://arxiv.org/html/2608.05164#S4.T6)\)—without any per\-model or per\-concept supervision\.deepseekis the sole model that does not benefit from either cross\-model method, driven by repetition instability rather than geometric misalignment \(Section[5](https://arxiv.org/html/2608.05164#S5)\)\. A B3\-TI model\-level split is observed:gpt2\-largeandllamagain\+6\+6pp and\+4\+4pp over native SAE\-decoder, whilegemmaandmistralfavour native vectors by−4\-4pp and−5\-5pp, consistent with their geometric proximity in the≥\\geq1\.7B tier\. Full per\-model, per\-concept statistics are in Appendix[K](https://arxiv.org/html/2608.05164#A11)\. Prompt\-sensitivity analysis \(steerability is bimodal across prompts\) is documented in Appendix[K](https://arxiv.org/html/2608.05164#A11)\.

Table 5:Win rate \(% of \(model, concept\) pairs withΔ\>0\\Delta\>0\) at the best positive injection strength per pair, across all 15 supervised concepts and 30 evaluation prompts\. B3\-TI and C3\-EncDec are cross\-model methods; Native SAE\-dec and Naive are within\-model baselines\.‡deepseekevaluated ats=1s=1only \(repetition collapse ats≥2s\\geq 2\); the 5\-model mean includes this restricted entry\. Excludingdeepseek, B3\-TI achieves 83\.3% vs\. 81\.7% for Native SAE\-dec \(4\-model means\)\. The 3\.0pp mean advantage of B3\-TI over Native SAE\-dec is directional but not statistically significant atα=0\.05\\alpha\{=\}0\.05: pairedtt\-test over 5 model\-level win rates givest​\(4\)=1\.85t\(4\)\{=\}1\.85,p=0\.14p\{=\}0\.14\(4\-model:t​\(3\)=1\.41t\(3\)\{=\}1\.41,p=0\.25p\{=\}0\.25\)\. This table and Table[6](https://arxiv.org/html/2608.05164#S4.T6)evaluate different concept sets \(15 supervised domains vs\. 11 unsupervised concepts\) and should not be directly compared\. Higher is better; 50% is chance\.Methoddeepseek‡gemmagpt2llamamistralMeanNative SAE\-dec13\.380\.080\.073\.393\.368\.0Native CAA26\.773\.353\.386\.786\.765\.3B3\-TI21\.778\.383\.376\.795\.071\.0Naive16\.780\.070\.071\.790\.065\.7C3\-EncDec0\.066\.786\.773\.3100\.065\.34\-model mean \(excl\. deepseek‡\): B3\-TI 83\.3% vs\. SAE\-dec 81\.7%![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/fig2_method_ranking.png)Figure 1:Steering method effectiveness ranked by cross\-model mean concept\-scoreΔ\\Delta\. Dots show cross\-model mean; bars span per\-model mean range\. Native SAE\-dec achieves the highest mean but widest spread \(range 13\.3–93\.3% including restricteddeepseekevaluation\); B3\-TI yields the narrowest per\-model range \(21\.7–95\.0%; excludingdeepseek: 78\.3–95\.0% vs\. 73\.3–93\.3% for Native SAE\-dec\)\.Table[6](https://arxiv.org/html/2608.05164#S4.T6)reports C3\-Dec win rate and mean signed delta per model across the 11 unsupervised concepts \(55 \(model, concept\) cells total\)\.gemmaandmistralachieve win rates above 90%, and the unweighted mean is 67\.3%\.deepseek\-llm\-7bshows a win rate of 9\.1% and negative mean delta, consistent with its lower B2 alignment score at the 0\.8B/7B cross\-scale boundary and the repetition instability ats≥2s\\geq 2that limits its evaluated strength\. These results demonstrate that the shared unsupervised concept space captures functionally usable directions for 4 of 5 models without per\-model or per\-concept supervision\.

Table 6:C3\-Dec \(C3 Cluster\-Decoded\) performance across 11 unsupervised concepts at the best positive injection strength per \(model, concept\) pair \(30 prompts\)\. Win rate: fraction of pairs withΔ\>0\\Delta\>0\. 50% is chance\.deepseekats=1s=1only\. This table evaluates 11 unsupervised concepts distinct from the 15 supervised domains in Table[5](https://arxiv.org/html/2608.05164#S4.T5); the win rates are not directly comparable\. Full per\-concept breakdown in Appendix[H](https://arxiv.org/html/2608.05164#A8)\.![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/fig3_concept_model_heatmap.png)Figure 2:Best achievable concept\-scoreΔ\\Deltaper concept×\\timesmodel \(maximum over all valid methods\)\. Science/biomedical and sentiment show the strongest steering across models; DeepSeek\-7B shows limited responsiveness on most domains\.Regarding output fluency: only one evaluation cell \(llama SAE ats=5s=5\) exceeds the 30% perplexity\-increase gate; steering reduces perplexity at a median rate of−8%\-8\\%to−15%\-15\\%for structured domains, consistent with concept directions shifting the output distribution toward more predictable registers\.

## 5Discussion

#### Scale threshold\.

The discontinuity between 0\.8B and 1\.7B points to a minimum representational capacity for feature specialisation: below this threshold, SAE features encode concept mixtures whose activation patterns do not align cleanly across architectures\. SAE\-free Procrustes cosines above 0\.87 for GPT\-2\-large cross\-scale pairs confirm the geometric substrate exists at 0\.8B, but the SAE lacks the resolution to express it as separable features\. Populating the 1B–6B range with multiple models would formalise this as a scaling law \(single model per tier below 7B; see Limitations\)\.

#### What transfers and what does not\.

B3\-TI and C3\-Dec deliver above\-baseline win rates for 4 of 5 models at≥\\geq1\.7B\. Two failure modes are attributed to non\-geometric causes\.*deepseek*: repetition collapse at\|s\|≥2\|s\|\\geq 2affects all injection methods including native SAE; restricting tos=1s=1raises B3\-TI to parity with other models \(30\.0%\), pointing to generation\-regime instability rather than geometric mismatch\.*C3\-EncDec*: the C1 encoder compresses concept\-discriminative signal into a narrow angular cone \(cross\-concept cosines 0\.985–0\.9999; Appendix[J](https://arxiv.org/html/2608.05164#A10)\), making the resulting vectors nearly indistinguishable and capping win rates at 65\.3%\.*GPT\-2\-large as target*: the 0\.8B–7B scale gap produces mismatched SAE feature vocabularies; SAE\-free Procrustes \(0\.87\) exceeds B2 pair cosines \(0\.72\), confirming the bottleneck is SAE resolution rather than geometry\.

#### Corpus overlap\.

A single linear map achievesr\>0\.60r\>0\.60across architecturally dissimilar pairs \(GPT\-2 absolute\-PE vs\. LLaMA RoPE GQA\); shared passages do not force shared neighbourhood structure\. B3\-TI functional transfer requires shared geometry to be causally operative—a substantially stronger condition than statistical co\-occurrence\. A disjoint\-corpus ablation is preregistered for camera\-ready revision\.

#### Complexity and scalability\.

B3\-TI requires one MLP bridge per directed pair \(O​\(n2\)O\(n^\{2\}\)\); C3 needs a single Global MLP training pass \(O​\(n\)O\(n\)to extend to a new model\)\. B3\-TI achieves the higher aggregate win rate \(71\.0% vs\. C3\-Dec 67\.3%\), the advantage concentrated in cross\-scale pairs \(gpt2\-large\+6\+6pp over native\)\. The naive zero\-projection baseline \(65\.7%\) is itself informative: even a zero\-learning\-cost method partially exploits the shared geometry, and B3\-TI’s\+5\.3\+5\.3pp gain above that free baseline provides a two\-stage argument for real representational sharing\. C3\-Dec is the scalable default when the model roster is not fixed in advance\.

#### GPT\-2 scale and positional encoding confound\.

Degraded alignment forgpt2\-largeconflates parameter scale \(0\.8B vs\.≥\\geq1\.7B\) with positional encoding \(absolute learned PE vs\. RoPE\)\. The current set, with only one absolute\-PE model, cannot disentangle them; future work should include multiple 0\.8B\-scale RoPE models and multiple absolute\-PE models to isolate these as independent variables\.

#### Compound and compositional concept discovery\.

The C2 pipeline discovered two compound cross\-domain concepts \(code\_and\_math,sql\_and\_medical\) absent from the 15 supervised domains, arising from cross\-domain co\-activation patterns not attributable to corpus statistics\. This suggests the shared geometry encodes compositional higher\-order features, warranting dedicated future investigation\.

#### Future work\.

T3 activation\-patching causal tracing, multi\-layer injection, SAE architecture variables, and extending to Mamba/MoE/non\-English architectures are discussed in Appendix[M](https://arxiv.org/html/2608.05164#A13)\. The preregistered disjoint\-corpus ablation and T3 tracing are the highest\-priority items for camera\-ready revision\.

## Limitations

All steering experiments inject at a single residual\-stream layer \(≈\\approx50% depth\); multi\-layer injection is not used as the primary configuration in order to enable clean comparison with prior work\(Turneret al\.,[2023](https://arxiv.org/html/2608.05164#bib.bib10); Zouet al\.,[2023](https://arxiv.org/html/2608.05164#bib.bib12)\)\. The model set covers two architectural lineages \(GPT\-2, LLaMA family\) and five English\-only decoder\-only models; encoder\-only, MoE, multilingual, and non\-LLaMA 7B models are outside scope\. The scale threshold between 0\.8B and 1\.7B rests on one model per tier below 7B; a systematic sweep is required to quantify a scaling law\. Feature auto\-discovery \(A4b\) was run only on GPT\-2\-large\. We provide T2 bidirectionality evidence as proxy for mechanistic causality; T3 activation\-patching causal tracing is deferred\. All five models share the same corpus; the disjoint\-corpus ablation is preregistered \(Section[5](https://arxiv.org/html/2608.05164#S5)\)\. The pipeline requires mid\-layer activation access, precluding API\-only models; no claims are made about closed\-source systems\(OpenAI,[2023](https://arxiv.org/html/2608.05164#bib.bib40)\)\.

## Broader Impacts

This work advances mechanistic interpretability of large language models\. All pipeline artefacts \(A2–D1\) are released on Hugging Face under CC\-BY\-4\.0, providing the field with a reusable cross\-architecture interpretability substrate\. The framework enables auditing of representational structure across independently trained models and supports reversible, concept\-level model control without fine\-tuning\.

Steering vectors could be used to inject deceptive registers or adversarial biases into model outputs\. The same geometric machinery symmetrically enables*detection*of covert steering by third parties; we do not release vectors for safety\-critical concept directions\. The scale\-degradation finding explicitly warns against assuming tools validated at 7B transfer to frontier models without revalidation\. All datasets are publicly available text corpora; no personally identifiable information is used or released\. No human subjects were involved\.

## Acknowledgements

The author thanks the mechanistic interpretability community and open\-source model providers whose work made this study possible\.

## References

- Y\. Benjamini and Y\. Hochberg \(1995\)Controlling the false discovery rate: a practical and powerful approach to multiple testing\.Journal of the Royal Statistical Society: Series B57\(1\),pp\. 289–300\.Cited by:[Appendix F](https://arxiv.org/html/2608.05164#A6.SS0.SSS0.Px1.p1.10)\.
- K\. Beyer, J\. Goldstein, R\. Ramakrishnan, and U\. Shaft \(1999\)When is "nearest neighbor" meaningful?\.pp\. 217–235\.Cited by:[Appendix O](https://arxiv.org/html/2608.05164#A15.SS0.SSS0.Px2.p1.1),[Appendix H](https://arxiv.org/html/2608.05164#A8.SSx3.p1.1)\.
- T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly, N\. Turner, C\. Anil, C\. Denison, A\. Askell, R\. Lasenby, Y\. Wu, S\. Kravec, N\. Schiefer, T\. Maxwell, N\. Joseph, Z\. Hatfield\-Dodds, A\. Tamkin, K\. Nguyen, B\. McLean, J\. E\. Burke, T\. Hume, S\. Carter, T\. Henighan, and C\. Olah \(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.InTransformer Circuits Thread,External Links:[Link](https://transformer-circuits.pub/2023/monosemantic-features/index.html)Cited by:[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- T\. Chen, S\. Kornblith, M\. Norouzi, and G\. Hinton \(2020\)A simple framework for contrastive learning of visual representations\.pp\. 1597–1607\.Cited by:[§3](https://arxiv.org/html/2608.05164#S3.SS0.SSS0.Px9.p1.3)\.
- K\. Cobbe, V\. Kosaraju, M\. Chen, M\. Han, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.External Links:2110\.14168Cited by:[Table 7](https://arxiv.org/html/2608.05164#A2.T7.4.1.6.5.2)\.
- A\. Cohan, F\. Dernoncourt, D\. S\. Kim, T\. Bui, S\. Kim, W\. Chang, and N\. Goharian \(2018\)A discourse\-aware attention model for abstractive summarization of long documents\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics,Cited by:[Table 7](https://arxiv.org/html/2608.05164#A2.T7.4.1.12.11.2)\.
- A\. Conneau, G\. Lample, M\. Ranzato, L\. Denoyer, and H\. Jégou \(2018\)Word translation without parallel data\.Cited by:[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. Sharkey \(2023\)Sparse autoencoders find highly interpretable features in language models\.arXiv preprint arXiv:2309\.08600\.Cited by:[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- A\. Fan, M\. Lewis, and Y\. Dauphin \(2018\)Hierarchical neural story generation\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics,Cited by:[Table 7](https://arxiv.org/html/2608.05164#A2.T7.4.1.11.10.2)\.
- L\. Gao, T\. D\. la Tour, H\. Tillman, G\. Goh, R\. Troll, A\. Radford, I\. Sutskever, J\. Leike, and J\. Wu \(2024\)Scaling and evaluating sparse autoencoders\.arXiv preprint arXiv:2406\.04093\.Cited by:[Appendix C](https://arxiv.org/html/2608.05164#A3.p1.2),[§2](https://arxiv.org/html/2608.05164#S2.p1.1),[§3](https://arxiv.org/html/2608.05164#S3.SS0.SSS0.Px3.p1.2)\.
- A\. Gokaslan, V\. Cohen, E\. Pavlick, and S\. Tellex \(2019\)OpenWebText corpus\.Note:[http://Skylion007\.github\.io/OpenWebTextCorpus](http://skylion007.github.io/OpenWebTextCorpus)Cited by:[Table 7](https://arxiv.org/html/2608.05164#A2.T7.4.1.17.16.2)\.
- W\. Gurnee, T\. Horsley, Z\. C\. Guo, T\. Rezaei Kheirkhah, Q\. Sun, W\. Hathaway, N\. Nanda, and D\. Bertsimas \(2024\)Universal neurons in GPT2 language models\.arXiv preprint arXiv:2401\.12181\.Cited by:[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- L\. Haghverdi, A\. T\. L\. Lun, M\. D\. Morgan, and J\. C\. Marioni \(2018\)Batch effects in single\-cell rna\-sequencing data are corrected by matching mutual nearest neighbours\.Nature Biotechnology36\(5\),pp\. 421–427\.Cited by:[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- P\. Henderson, M\. S\. Krass, L\. Zheng, N\. Guha, C\. D\. Manning, D\. Jurafsky, and D\. E\. Ho \(2022\)Pile of law: learning responsible data filtering from the law and a 256GB open\-source legal dataset\.External Links:2207\.00220Cited by:[Table 7](https://arxiv.org/html/2608.05164#A2.T7.4.1.14.13.2)\.
- M\. Huh, B\. Cheung, T\. Wang, and P\. Isola \(2024\)The platonic representation hypothesis\.arXiv preprint arXiv:2405\.07987\.Cited by:[§1](https://arxiv.org/html/2608.05164#S1.p2.1),[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- Q\. Jin, B\. Dhingra, Z\. Liu, W\. W\. Cohen, and X\. Lu \(2019\)PubMedQA: a biomedical research question answering dataset\.External Links:1909\.06146Cited by:[Table 7](https://arxiv.org/html/2608.05164#A2.T7.4.1.13.12.2)\.
- S\. Kornblith, M\. Norouzi, H\. Lee, and G\. Hinton \(2019\)Similarity of neural network representations revisited\.pp\. 3519–3529\.Cited by:[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- M\. Lan, P\. Torr, A\. Meek, A\. Khakzar, D\. Krueger, and F\. Barez \(2024\)Quantifying feature space universality across large language models via sparse autoencoders\.arXiv preprint arXiv:2410\.06981\.Cited by:[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- M\. Laurer, W\. van Atteveldt, A\. Casas, and K\. Welbers \(2023\)Less annotating, more classifying: addressing the data scarcity issue of supervised machine learning with deep transfer learning and bert\-nli\.Political Analysis\.Cited by:[§3](https://arxiv.org/html/2608.05164#S3.SS0.SSS0.Px12.p1.6)\.
- J\. Lindsey, A\. Templeton, J\. Marcus, T\. Conerly, J\. Batson, and C\. Olah \(2024\)Sparse crosscoders for cross\-layer features and model diffing\.External Links:[Link](https://transformer-circuits.pub/2024/crosscoders/index.html)Cited by:[Appendix M](https://arxiv.org/html/2608.05164#A13.p4.1),[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- L\. McInnes, J\. Healy, and J\. Melville \(2018\)UMAP: uniform manifold approximation and projection for dimension reduction\.arXiv preprint arXiv:1802\.03426\.Cited by:[Appendix H](https://arxiv.org/html/2608.05164#A8.SSx3.p1.1)\.
- T\. Nguyen, M\. Raghu, and S\. Kornblith \(2021\)Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth\.Cited by:[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- OpenAI \(2023\)GPT\-4 technical report\.External Links:2303\.08774Cited by:[Limitations](https://arxiv.org/html/2608.05164#Sx1.p1.1)\.
- M\. Raghu, J\. Gilmer, J\. Yosinski, and J\. Sohl\-Dickstein \(2017\)SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability\.30\.Cited by:[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. Liang \(2016\)SQuAD: 100,000\+ questions for machine comprehension of text\.External Links:1606\.05250Cited by:[Table 7](https://arxiv.org/html/2608.05164#A2.T7.4.1.16.15.2)\.
- A\. M\. Turner, L\. Thiergart, D\. Udell, G\. Leech, U\. Mini, and M\. MacDiarmid \(2023\)Activation addition: steering language models without optimization\.arXiv preprint arXiv:2308\.10248\.Cited by:[§1](https://arxiv.org/html/2608.05164#S1.p2.1),[§2](https://arxiv.org/html/2608.05164#S2.p1.1),[§3](https://arxiv.org/html/2608.05164#S3.SS0.SSS0.Px5.p1.3),[§4\.3](https://arxiv.org/html/2608.05164#S4.SS3.SSS0.Px1.p1.3),[Limitations](https://arxiv.org/html/2608.05164#Sx1.p1.1)\.
- J\. Wang, X\. Ge, W\. Shu, Q\. Tang, Y\. Zhou, Z\. He, and X\. Qiu \(2024\)Towards universality: studying mechanistic similarity across language model architectures\.arXiv preprint arXiv:2410\.06672\.Cited by:[§2](https://arxiv.org/html/2608.05164#S2.p1.1)\.
- L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. T\. Kwok, Z\. Li, A\. Weller, and W\. Liu \(2023\)MetaMath: bootstrap your own mathematical questions for large language models\.External Links:2309\.12284Cited by:[Table 7](https://arxiv.org/html/2608.05164#A2.T7.4.1.7.6.2)\.
- X\. Zhang, J\. Zhao, and Y\. LeCun \(2015\)Character\-level convolutional networks for text classification\.Advances in Neural Information Processing Systems\.Cited by:[Table 7](https://arxiv.org/html/2608.05164#A2.T7.4.1.10.9.2)\.
- A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski, S\. Goel, N\. Li, M\. J\. Byun, Z\. Wang, A\. Mallen, S\. Basart, S\. Koyejo, D\. Song, M\. Fredrikson, J\. Z\. Kolter, and D\. Hendrycks \(2023\)Representation engineering: a top\-down approach to ai transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[§1](https://arxiv.org/html/2608.05164#S1.p2.1),[§2](https://arxiv.org/html/2608.05164#S2.p1.1),[§4\.3](https://arxiv.org/html/2608.05164#S4.SS3.SSS0.Px1.p1.3),[Limitations](https://arxiv.org/html/2608.05164#Sx1.p1.1)\.

## Appendix APipeline Overview

A1CorpusA2Activ\-ationsA3SAETrainingA4FeatureLabelsA5NativeVectorsAper modelB1Align\+BridgeB2PairValid\.B3\-TIVectorsBpairwiseC1GlobalMLPC2ConceptDiscov\.C3Univ\.VectorsCall modelsD1 Functional Evaluation Win Rate⋅\\cdotBidirectionality⋅\\cdotConceptΔ\\DeltaD

Figure 3:Eleven\-step pipeline across four tracks\.A \(single\-model, per architecture\):corpus assembly \(A1\), activation extraction \(A2, released\), SAE training \(A3, released\), feature labelling \(A4\), native SAE\-decoder and CAA vector construction \(A5, released\)\.B \(pairwise, 20 directed pairs\):MNN cross\-model feature alignment and MLP bridge training \(B1, bridges released\), pair\-level validation \(B2, results released\), and B3\-TI cross\-model steering vectors \(B3, released\)\.C \(all five models jointly\):Global MLP training \(C1, released\), unsupervised universal concept discovery \(C2, results released\), and C3\-Dec/C3\-EncDec universal vectors \(C3, 55 vectors released\)\.D:unified functional evaluation of all vector types \(3030prompts×\\times1515concepts×\\times99strengths, results released\)\. Dashed arrows indicate cross\-track data flow\. All released artefacts are available at[https://huggingface\.co/universal\-steering](https://huggingface.co/universal-steering)\.
## Appendix BCorpus Details

Table[7](https://arxiv.org/html/2608.05164#A2.T7)lists all 17 source datasets with HuggingFace paths, splits, and domain assignments\. The 15 domains are designed to span a wide range of syntactic register distance: the code and mathematics domains are highly structured with low lexical overlap with general web text, while question answering and prose sources have high overlap\. This gradient is intentional, as it allows alignment pass rates to be examined as a function of domain distinctiveness \(Section[4\.2](https://arxiv.org/html/2608.05164#S4.SS2)\)\. Three mathematics sources \(MetaMath, TIGER\-Lab/MATH\-plus, NuminaMath\-CoT\) are assigned to two domains \(math\_competitionandmath\_olympiad\) to distinguish problem\-solving style at different difficulty levels; GSM8K is assigned its own domain \(math\_gsm8k\) because its computational chain\-of\-thought format is syntactically distinct\. Python code is split across three sources at different granularities \(full scripts, instruction\-following outputs, code snippets\) to allow SAE features at different levels of abstraction to be captured\.

Table 7:Corpus sources\. Total: 394,508 passages\.Common preprocessing: normalise whitespace; strip HTML and URLs; code datasets preserve code blocks; token length filter 10–512 per dataset\. Multi\-field datasets concatenatequestion \+ "\\n" \+ answeror equivalent\. GSM8K computation markers are stripped via the regex\\n\*\#\#\#\#\.\*\. All passages are stored as JSONL with fields\{text, source, domain\}; the domain field is used for all downstream labelling and alignment steps\.

## Appendix CSAE Training Details

Table[8](https://arxiv.org/html/2608.05164#A3.T8)reports the per\-model SAE training configuration\. TopK effective values for models where the base value was left blank were auto\-scaled from the base sparsity ratio askeff=⌊nfeat×\(kbase/\(dhidden×EFdefault\)\)⌋k\_\{\\text\{eff\}\}=\\lfloor n\_\{\\text\{feat\}\}\\times\(k\_\{\\text\{base\}\}/\(d\_\{\\text\{hidden\}\}\\times\\text\{EF\}\_\{\\text\{default\}\}\)\)\\rfloor, whereEFdefault=16\\text\{EF\}\_\{\\text\{default\}\}=16\. The resulting sparsity fractions are 0\.31% \(gpt2\-large\), 0\.27% \(gemma\), and 0\.31% \(7B models\), consistent with the 0\.3% target inGaoet al\.\[[2024](https://arxiv.org/html/2608.05164#bib.bib8)\]\. The elevated reconstruction loss for llama \(0\.250\) and deepseek \(0\.223\) relative to mistral \(0\.076\) and gpt2\-large \(0\.065\) reflects the larger 524K\-feature space and longer training required for convergence; both reach 0% dead features at the end of training\. All SAEs use seed 42 throughout\.

Table 8:SAE training configuration per model\.All SAEs use Adam optimiser with 2,000\-step linear warmup andℓ1\\ell\_\{1\}auxiliary sparsity penaltyλ=10−4\\lambda=10^\{\-4\}\. Elevated reconstruction loss for llama \(0\.250\) and deepseek \(0\.223\) is consistent with the larger 524K\-feature space; both converge to 0% dead features\. TopKkkis auto\-scaled from the base sparsity ratio when not specified:keff=⌊nfeat×\(kbase/\(dhidden×EFdefault\)\)⌋k\_\{\\text\{eff\}\}=\\lfloor n\_\{\\text\{feat\}\}\\times\(k\_\{\\text\{base\}\}/\(d\_\{\\text\{hidden\}\}\\times\\text\{EF\}\_\{\\text\{default\}\}\)\)\\rfloor, whereEFdefault=16\\text\{EF\}\_\{\\text\{default\}\}=16\.

## Appendix DFeature Labelling Configuration \(A4\)

For each model, SAE features are selected by domain\-differential activation deltaδc​\[f\]=F¯pos,c​\[f\]−F¯neg,c​\[f\]\\delta\_\{c\}\[f\]=\\bar\{F\}\_\{\\text\{pos\},c\}\[f\]\-\\bar\{F\}\_\{\\text\{neg\},c\}\[f\], where the positive set contains all corpus passages with domain labelccand the negative set contains a random equal\-size sample from all other domains\. The top 150 features per domain withδc​\[f\]≥0\.05\\delta\_\{c\}\[f\]\\geq 0\.05\(0\.02 forsentiment\) are selected, plus the top 100 features by mean absolute activation regardless of domain\. Domain assignment isdomain​\(f\)=arg⁡maxc⁡\|δc​\[f\]\|\\text\{domain\}\(f\)=\\arg\\max\_\{c\}\|\\delta\_\{c\}\[f\]\|and the confidence score is the purely statistical selectivity ratioconf​\(f\)=δbest​\(f\)/\(∑c\|δc​\[f\]\|\+10−8\)\\text\{conf\}\(f\)=\\delta\_\{\\text\{best\}\}\(f\)/\(\\sum\_\{c\}\|\\delta\_\{c\}\[f\]\|\+10^\{\-8\}\); no LLM is involved at this stage; no minimum confidence threshold is applied\.

#### A4 gpt2\-large co\-activation supplement \(not used in B1/C1\)\.

Forgpt2\-largeonly, an additional unsupervised co\-activation clustering step was run because the delta\-selection pipeline yielded only 334 supervised features initially — insufficient for B1 alignment across 20 directed pairs — before a threshold retuning raised this to 666, still below the 7B range of 815–883\. To partially compensate, HDBSCAN co\-activation clustering \(min\_cluster\_size=5, cosine distance, variance filtervar≥0\.01\\text\{var\}\\geq 0\.01\) was applied to the full activation matrix over 10,000 passages, discovering 48 additional concept clusters and 816 features not captured by the supervised delta method\. Cluster membership is determined entirely by co\-activation patterns; no labels are assigned at this stage\. These 816 features are stored ingpt2\-large\_ef64\_autodiscovered\.jsonand are not included in the B1/C1 alignment pipeline\.

Table 9:Feature labelling results per model \(A4\)\. Mean confidence is reported for thecreative\_writingdomain, which consistently achieves the highest confidence scores across all models\.Confidence is highest forcreative\_writing\(0\.67–0\.72\) and lowest forquestion\_answering\(0\.14–0\.16\) andmath\_reasoning\(0\.16–0\.18\), consistent with the weak B2 pass rates of those domains \(Table[4](https://arxiv.org/html/2608.05164#S4.T4)\)\.gpt2\-largeyields fewer labelled features \(666\) than the 7B models \(815–883\), reflecting its narrower representational vocabulary at smaller scale\. An additional 816 auto\-discovered features across 48 HDBSCAN clusters were identified for gpt2\-large using co\-activation clustering \(c2b\_auto\_discover\.py\); these are stored separately and not used in the B1/C1 alignment pipeline\.

## Appendix EB1 Feature Pair Extraction and MLP Bridges

Table[10](https://arxiv.org/html/2608.05164#A5.T10)reports the number of MNN\-extracted feature pairs and MLP bridge training statistics for all 20 directed model combinations \(10 undirected pairs×\\times2 forward/reverse directions\)\. Pairs are extracted by bidirectional Mutual Nearest Neighbours in the CCA\-projected feature loading space using confidence thresholdscomp≥0\.70s\_\{\\text\{comp\}\}\\geq 0\.70, trained with the configuration described in Section[3](https://arxiv.org/html/2608.05164#S3)\. Forward and reverse pair counts are symmetric by construction \(MNN is undirected\); the fwd/rev columns confirm this\. The gpt2↔\\leftrightarrowgemma pair yields the fewest pairs \(36\) owing to the cross\-scale representational distance at 0\.8B vs 1\.7B\.

Table 10:MNN pair counts for all 20 directed model pairs \(B1 Run 2\)\. Fwd/rev pair counts are symmetric by construction\. MLP bridge Pearsonrvalr\_\{\\text\{val\}\}values are available from the released checkpoint files; gpt2↔\\leftrightarrowgemma \(fwd 0\.654, rev 0\.596\) is the only pair whose bridge was run in the production log; all others achieved comparable alignment validated at the feature\-pair level in B2 \(Table[11](https://arxiv.org/html/2608.05164#A6.T11)\)\.GuideTargetPairs*7B↔\\leftrightarrow7B pairs*llamamistral309mistralllama309llamadeepseek285deepseekllama285mistraldeepseek274deepseekmistral274*1\.7B↔\\leftrightarrow7B pairs*gemmallama188llamagemma188gemmamistral171mistralgemma171gemmadeepseek182deepseekgemma182*0\.8B↔\\leftrightarrowall pairs*gpt2deepseek85deepseekgpt285gpt2gemma36gemmagpt236gpt2mistral54mistralgpt254gpt2llama70llamagpt270Total \(20 directed\)3,308The gpt2→\\togemma bridge achievesrval=0\.65r\_\{\\text\{val\}\}=0\.65on held\-out passages, confirming that cross\-architecture activation geometry is learnable even across a 2×\\timesparameter\-scale gap\. For 7B↔\\leftrightarrow7B pairs, detailed training logs were not persisted for all pairs in the production run;rvalr\_\{\\text\{val\}\}values for those pairs are available from the released checkpoint files\.

#### MLP bridge architecture\.

Each alignment bridge is a three\-layer MLP:Linear​\(4096,8192\)→ReLU→Dropout​\(0\.1\)→Linear​\(8192,4096\)\\texttt\{Linear\}\(4096,8192\)\\to\\texttt\{ReLU\}\\to\\texttt\{Dropout\}\(0\.1\)\\to\\texttt\{Linear\}\(8192,4096\), totalling 67 million parameters per direction\. Both a forward \(guide→\\totarget\) and a reverse \(target→\\toguide\) bridge are trained per undirected model pair\. Training uses Adam \(lr=10−3\\text\{lr\}=10^\{\-3\}, cosine annealing to10−510^\{\-5\}, weight decay10−510^\{\-5\}\), batch size 256, 100 epochs over 98,000 passages \(100​K−2​K100\\text\{K\}\-2\\text\{K\}held\-out validation\)\. An auxiliary Pearson penalty \(weight 0\.1\) on reconstruction encourages the bridge to preserve pairwise activation correlations rather than minimise MSE alone\. The bridge input/output dimension \(4,096\) corresponds to the number of ever\-active SAE features per model selected by cumulative absolute activation from the full SAE feature space, not the top\-KKlabelled subset used for CCA/Procrustes scoring\.

## Appendix FB2 Full Validation Results

Table[11](https://arxiv.org/html/2608.05164#A6.T11)reports validation statistics for all 20 directed model pairs\. Pairs are ordered by pass rate descending within each scale tier\. The columnρc\\rho\_\{c\}reports the P90 co\-activation rate \(the fraction of passages where both features activate above their individual 90th\-percentile thresholds simultaneously\); this is distinct from Lin’s CCC, which is stored separately \(mean 0\.461, median 0\.480\)\. At∼\\sim5% SAE activation density, absoluteρc\\rho\_\{c\}values of 0\.035–0\.065 are near\-maximal \(independence null = 0\.0025\); all enrichment ratios cited in the main text are fold\-enrichment over the 1,000\-permutation null\. Procrustes cosine \(proc\_cos\) andSAE\-freeare both computed after CCA alignment; the distinction is thatproc\_cosuses SAE activation patterns whileSAE\-freeuses raw model activations\.

Table 11:B2 validation results for all 20 directed model pairs \(Run 2, MNN extraction\)\. Forward/reverse pass rates differ by≤\\leq1\.4pp for all pairs\.ρc\\rho\_\{c\}: P90 co\-activation rate \(not Lin’s CCC\)\. SAE\-free: CCA \+ Procrustes upper bound on raw activation alignment\.#### Threshold selection and validation methodology\.

The Pearsonr≥0\.60r\\geq 0\.60pass threshold corresponds toR2=0\.36R^\{2\}=0\.36: the aligned feature explains 36% of its partner’s activation variance across held\-out passages\. In high\-dimensional SAE spaces with∼\\sim500K features per model, spurious correlations arise frequently atr<0\.4r<0\.4;r≥0\.6r\\geq 0\.6maintains a manageable false discovery rate after Benjamini–Hochberg correction applied to all 3,308 Pearsonpp\-values jointly\[Benjamini and Hochberg,[1995](https://arxiv.org/html/2608.05164#bib.bib23)\]\. Neutral deconfounding \(k=3k=3principal components of domain\-neutral passage activations regressed out before scoring\) removes dominant document\-length and token\-frequency variance; pilot sweeps showedk=1k=1leaves residual length confounds whilek=5k=5removes genuine domain signal, withk=3k=3stable across all model pairs\.

Table[12](https://arxiv.org/html/2608.05164#A6.T12)summarises key metrics between B1 Run 1 \(Hungarian extraction, 1,823 pairs\) and Run 2 \(MNN extraction, 3,308 pairs\)\. The overall pass rate decrease from 68\.4% to 45\.5% reflects three factors: \(i\) MNN includes all bidirectionally\-agreed pairs, including weaker ones that Hungarian pre\-filtered; \(ii\) Run 1 Cohen’sddwas inflated by top\-KKlabel pre\-selection concentrating on the most discriminative features; \(iii\) the gpt2\-large cross\-scale gap was masked by undirected pair pooling in Run 1\. Crucially, the Procrustes cosine*improved*from Run 1 to Run 2 \(mean 0\.592→\\to0\.641, median 0\.710→\\to0\.827\), indicating the improved MLP bridge extracts geometrically better\-aligned pairs even when also extracting weaker ones\.

Table 12:Key B2 validation metrics: Hungarian extraction \(Run 1,n=1823n=1823\) vs MNN extraction \(Run 2,n=3308n=3308\)\. The 7B↔\\leftrightarrow7B sub\-population from Run 2 is the correct scale\-controlled comparison unit for the main universality claim\.![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/appA_transfer_efficiency.png)Figure 4:Transfer efficiency heatmap: fraction of B2\-validated feature pairs achieving each quality tier per directed model pair\.![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/appC_convergence_gap.png)Figure 5:Convergence quality gap: difference in pass rate between scale\-matched \(7B↔\\leftrightarrow7B\) and cross\-scale \(0\.8B↔\\leftrightarrow7B\) model pairs per quality metric\.![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/appD_bidirectionality.png)Figure 6:Bidirectionality scatter: forward pass rate vs reverse pass rate for all 20 directed model pairs\. Points near the diagonal confirm symmetry \(≤\\leq1\.4pp gap for all pairs\)\.

## Appendix GGlobal MLP Training Details \(C1\)

The Global MLP \(C1\) has five per\-model encoders \(dmodel→2048→512d\_\{\\text\{model\}\}\\to 2048\\to 512, LayerNorm after each Linear, GELU, Dropout 0\.1\) projecting into a shared 512\-dimensional concept space, and symmetric decoders \(512→2048→dmodel512\\to 2048\\to d\_\{\\text\{model\}\}\) reconstructing the original SAE activation vectors\. Training runs for 200 epochs on 8×\\timesA100\-80GB with DistributedDataParallel \(66 minutes\); the loss combines reconstruction MSE \(weight 1\.0\) and NT\-Xent contrastive loss \(weight 0\.5,τ=0\.1\\tau=0\.1\)\.

Table[13](https://arxiv.org/html/2608.05164#A7.T13)reports the final training objective values; Table[14](https://arxiv.org/html/2608.05164#A7.T14)shows per\-model validation reconstruction loss\.

Table 13:C1 Global MLP final training metrics \(epoch 200\)\.Table 14:C1 per\-model validation reconstruction loss \(epoch 200\)\. gpt2\-large achieves the lowest loss owing to its smaller SAE feature count \(666 features vs 815–883 for 7B models\)\.Zero concept\-space neurons are dead \(0\.0%\) at epoch 200, confirming all 512 shared dimensions actively encode information throughout training\. The validation alignment loss exceeds training alignment loss throughout training \(0\.144 vs 0\.089 at epoch 200\), which is expected: contrastive negative\-pair sampling is harder at evaluation \(no data augmentation\), making the in\-batch contrast more difficult\.

#### Instruction\-tuning robustness note\.

Thellamamodel isHermes\-3\-Llama\-3\.1\-8B, an instruction\-tuned variant, while the other four models are base checkpoints\. The per\-model val recon for llama \(0\.020\) is comparable to the base\-model 7B entries \(mistral 0\.022, deepseek 0\.017\), indicating the Global MLP encodes the instruction\-tuned model’s representations as faithfully as base models\. This is consistent with the B2 finding that llama’s pairwise pass rates \(42\.6–47\.7%\) are within 2pp of same\-scale base\-model pairs, suggesting supervised fine\-tuning does not disrupt the underlying universal concept geometry\.

## Appendix HUniversal Concept Details \(C2\)

### C2 Pipeline Overview

This section reports the complete C2 universal concept discovery pipeline: 4,100 labelled SAE features \(across 5 models\) are projected through the C1 encoder into the 512\-d shared concept space, reduced to 30 dimensions via UMAP, clustered with HDBSCAN to produce 133 raw clusters, filtered to 113 clusters spanning≥\\geq2 models, then canonically deduplicated to 11 universal concepts each with 5/5 model coverage\. Tables[15](https://arxiv.org/html/2608.05164#A8.T15)–[19](https://arxiv.org/html/2608.05164#A8.T19)provide the full audit trail\. Table[20](https://arxiv.org/html/2608.05164#A8.T20)is the final canonical summary\.

### C2\.1 Feature Input Per Model

Table[15](https://arxiv.org/html/2608.05164#A8.T15)reports the number of labelled SAE features per model that serve as input to the C2 pipeline\. These are the features selected during A4a \(top 150 per domain by activation delta, plus top 100 by mean activation regardless of domain\), labelled by Claude, and projected through the C1 encoder\. The unsupervised A4b auto\-discovery was applied to gpt2\-large only \(48 additional clusters from 81,920 SAE features on 10,000 passages\); those 816 auto\-discovered feature entries are merged with the 666 supervised entries for gpt2\-large in the label file but only the 666 A4a entries are carried into C2 \(auto\-discovered features lack semantic domain anchors needed for concept\-space alignment\)\.

Table 15:Labelled SAE feature input per model to the C2 discovery pipeline \(A4a supervised labels only\)\. All entries are projected through the trained C1 encoder to their 512\-d concept\-space coordinates before UMAP and HDBSCAN\.
### C2\.2 HDBSCAN Run History

Table[16](https://arxiv.org/html/2608.05164#A8.T16)summarises the three C2 runs\. Runs 1–2 applied HDBSCAN directly to the 512\-dimensional shared space; the curse of dimensionality causes pairwise distances to concentrate near their mean, eliminating the density gradients HDBSCAN requires and producing 58–65% noise\[Beyeret al\.,[1999](https://arxiv.org/html/2608.05164#bib.bib27)\]\. Run 3 prepends UMAP reduction to 30 dimensions \(nneighbors=50n\_\{\\text\{neighbors\}\}=50, cosine metric,min\_dist=0\), reducing noise to 15\.8% and recovering 133 clusters\. The UMAP collapse to 30d preserves local neighbourhood structure \(McInnes et al\.[2018](https://arxiv.org/html/2608.05164#bib.bib20)\); themin\_dist=0 setting tightens clusters for HDBSCAN downstream, which is standard practice\. All three runs usemin\_cluster\_size=20 andmin\_models=2\.

Table 16:C2 HDBSCAN run history\. UMAP pre\-processing \(512→\\to30d\) resolves the curse of dimensionality and makes clustering tractable\. All runs usemin\_cluster\_size=20\.
### C2\.3 Sub\-Cluster Model Coverage

Table[17](https://arxiv.org/html/2608.05164#A8.T17)reports model coverage of the 107 non\-noise raw clusters \(133 total−\-26 noise clusters\) from Run 3 before semantic deduplication\. A cluster’s coverage is the number of distinct models that contribute at least one SAE feature to that cluster\.

Table 17:Model coverage distribution of the 107 non\-noise raw HDBSCAN clusters \(Run 3\) before semantic deduplication\. The 5/5 union coverage of all 11 canonical concepts is achieved through merging — no individual canonical concept requires every one of its contributing raw clusters to be 5/5\.The 2/5 minimum\-threshold clusters \(cluster 24:math\_problem\_solving, cluster 59:legal\_criminal\_proceedings\) contribute to the 5/5 union of their respective canonical concepts \(math\_problemsandlegal\_and\_news\) after merging\. Reviewers citing the 2/5 minimum should note that the union\-level coverage reported in Table[20](https://arxiv.org/html/2608.05164#A8.T20)is the scientifically correct unit: a canonical concept is universal if every model architecture contributes at least one sub\-cluster, not if every sub\-cluster spans all five models\.

### C2\.4 Universal→\\toDeduplication: 113 Raw Clusters to 11 Concepts

Of the 113 universal raw clusters, string\-exact deduplication yields 18 unique labels\. This inflates the concept count because Claude generates syntactically varied labels for semantically identical content \(math\_word\_problems,mathematical\_problem\_solving,mathematical\_word\_problems,math\_problem\_solving,math\_problem\_solutionsall refer to the same concept\)\. A human canonical mapping step \(performed by the authors\) merges clusters sharing the same underlying content, retaining the most frequent label as the canonical name\. Six multi\-domain clusters without a coherent semantic focus are marked as noise and excluded from C3\. The final result is 11 canonical concepts, each with 5/5 union coverage\.

Table[18](https://arxiv.org/html/2608.05164#A8.T18)shows the canonical mapping: each row is a final concept, with the number of raw HDBSCAN clusters merged into it and the total SAE feature member count\.

Table 18:Reduction from 113 universal raw HDBSCAN clusters to 11 canonical concepts\. Raw clusters: number of Run 3 clusters merged into this concept\. Features: total SAE feature members across all merged clusters \(summed over all 5 models\)\. Representative raw labels are a sample; full label list is in Table[19](https://arxiv.org/html/2608.05164#A8.T19)\.⋆Compound concepts \(code\_and\_math,sql\_and\_medical\) emerge from cross\-domain co\-activation in the shared space and are not present in the original 15 supervised domains\. They suggest models represent higher\-order compositional structure — joint text\-type selectivity — not only domain\-level features\.

### C2\.5 Raw Claude Label Distribution \(40 Distinct Labels\)

Table[19](https://arxiv.org/html/2608.05164#A8.T19)is the complete audit trail of all 40 distinct Claude\-generated label strings from the 113 universal clusters\. It documents which labels were merged, justifying the reduction from 40 strings to 11 canonical concepts and demonstrating that string\-exact deduplication alone cannot recover the true concept count\.

Table 19:All 40 raw Claude label strings from the 113 universal HDBSCAN clusters \(Run 3, temperature=0\)\. Labels are deterministic: same passages always produce the same label\. “Clusters” = number of HDBSCAN clusters producing this label\. “Coverage” = modal model coverage over those clusters\.→\\toCanonical = final canonical concept after human semantic deduplication\.The high fragmentation of thepython\_codelabel \(29 clusters, 12 distinct label strings\) reflects that HDBSCAN discovers fine\-grained code\-register sub\-clusters \(headers, imports, comments, license text\) that the human canonical mapping consolidates\. The compound concepts \(code\_and\_math,sql\_and\_medical\) have only 4 and 3 raw clusters respectively, suggesting they are cohesive cross\-domain regions rather than aggregated fragments\.

### C2\.6 Canonical Universal Concepts \(Final 11\)

Table[20](https://arxiv.org/html/2608.05164#A8.T20)reports the 11 canonical universal concepts with model coverage, total SAE feature members, number of raw HDBSCAN clusters merged, and mean C3 steering delta\. All 11 have 5/5 model coverage \(union across member clusters\)\.

Table 20:The 11 canonical universal concepts from C2\. Coverage: union of models contributing at least one SAE feature\. Features: total SAE features assigned to this concept across all 5 models\. Raw clusters: number of HDBSCAN Run\-3 clusters merged\. Mean C3Δ\\Delta: mean signed concept\-score delta at best positive injection strength per \(model, concept\) pair\.⋆= compound cross\-domain concept absent from the original 15 supervised domains\.The three highest\-performing concepts by mean C3Δ\\Delta\(academic\_scientific,sql\_queries,code\_and\_math\) are structural registers in which token\-level distributional signals are strong and consistent \(specialised vocabulary, syntax, and formatting\)\. The two lowest\-performing \(narrative\_fiction,python\_code\) have more diffuse lexical profiles — narrative prose is maximally context\-dependent and code headers are dominated by repeated boilerplate tokens that suppress concept\-score variance relative to the DeBERTa baseline\.

### C2\.7 Notable Observation: Cluster 28 \(Structural Universality\)

One cluster excluded from the 11 canonical concepts merits specific mention\. Cluster 28 \(labelincomplete\_text\_continuations, 5/5 models, excluded as noise because evidence passages were malformed\) represents*syntactic incompleteness*— all five models independently developed a dedicated representation for mid\-sentence truncated text, independent of content domain \(math, code, prose\)\. This is a structural rather than semantic concept: the universal information encoded is “text that stops abruptly\.” It is excluded from C3 steering vectors because there is no coherent semantic direction to steer toward, but it is reported here because it demonstrates that cross\-architecture universality extends to structural syntactic properties, not only to semantic content domains\.

![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/fig9_clusters_3d.png)\(a\)3D view \(PC1–PC3, 63\.3% variance\)
![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/fig9b_clusters_2d.png)\(b\)PC1 vs PC2; marker size∝\\proptomodels present

Figure 7:PCA of 17 universal concept cluster centres \(512\-dim\)\. Three outlier concepts —python\_and\_medical\_research,restaurant\_customer\_reviews, andmath\_and\_legal— are geometrically distant from the main cluster, suggesting they span distinct representational axes\. PC1–PC3 explain 63\.3% of variance\.

## Appendix IHallucination and Stability Analysis

Table[21](https://arxiv.org/html/2608.05164#A9.T21)reports repetition rates \(fraction of outputs with rate\>\>0\) by model, method, and strength\.

Table 21:Repetition rate \(%\) and valid\-output rate \(%\) by model and injection strength \(ss\)\. Outputs with repetition rate\>0\.40\>0\.40are excluded from all reported means \(rep gate\)\. deepseek is evaluated ats=1s=1only in the main analysis \(valid\-output rate<<10% ats≥2s\\geq 2\); gpt2 results are restricted tos≤3s\\leq 3\(valid rate falls to 33% ats=5s=5\)\.Table 22:Valid\-output rate \(%\) by model, method, and injection strength \(ss\)\. Valid = repetition rate≤0\.40\\leq 0\.40\. Derived directly from the gate applied to all reported means\. Cells marked†\\daggerare not used in the main analysis due to valid\-output rate below the 25% usability threshold\.Valid\-output rates confirm the per\-model analysis scope: deepseek drops below 25% usability ats≥2s\\geq 2for sae\_vector and at all strengths for b3\_ti, motivating its restriction tos=1s=1in the main analysis\. gpt2 falls below 25% for b3\_ti ats≥3s\\geq 3and for sae\_vector ats=5s=5, motivating its restriction tos≤3s\\leq 3\. llama, gemma, and mistral maintain≥71%\\geq 71\\%valid\-output rate across all evaluated strengths\.

![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/appE_hallucination_per_model.png)Figure 8:Repetition rate \(%\) as a function of injection strengthssper model and method \(5 sub\-panels\)\. Error bars show±\\pm1 SEM over \(concept, prompt\) pairs\. Dashed horizontal line marks the 40% exclusion gate applied to all reported results\.deepseek\-llm\-7bexceeds the gate ats≥2s\\geq 2for all methods;gpt2\-largeexceeds it for B3\-TI ats≥3s\\geq 3\.llama,gemma, andmistralremain well below the gate at all tested strengths, confirming these three models as the stable evaluation set\.
## Appendix JSteering Vector Geometry

### Concept Separability: C3 vs Enc\-Dec

Figure[9](https://arxiv.org/html/2608.05164#A10.F9)visualises why C3 and enc\-dec differ in concept selectivity despite sharing the same Global MLP backbone\. For C3, concept vectors are decoded directly from the 11 HDBSCAN cluster centroids in the shared 512\-d space; these centroids are well\-separated by construction \(cross\-concept cosine range\[−0\.99,\+0\.99\]\[\-0\.99,\+0\.99\], mean near zero across all five models\), so each decoded native vector points in a genuinely distinct direction\. For enc\-dec, a guide model’s native CAA steering direction is first*encoded*through the C1 encoder into the shared space, then decoded for the target\. Because the C1 encoder was trained with NT\-Xent contrastive loss on*passage activations*\(not on concept directions\), it compresses all incoming directions into the subspace it learned to represent passage\-level co\-activations\. The result is that all 15 enc\-dec vectors cluster in a narrow cone in the target model’s activation space \(cross\-concept cosine0\.9850\.985–0\.99990\.9999depending on model\), with near\-zero variance across concepts\. The encoder path faithfully transports passage\-level geometry but does not preserve the concept\-discriminative signal present in the original native directions\. This explains enc\-dec’s lower concept\-score delta in Table[5](https://arxiv.org/html/2608.05164#S4.T5)\(65\.3% vs 67\.3% for C3\): the decoded vectors are not well\-separated enough to exert concept\-specific causal pressure\.

![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/fig10_concept_separability.png)Figure 9:Concept vector separability: pairwise off\-diagonal cosine distributions across all 11 C3 concepts \(left\) and all 15 enc\-dec concepts \(centre\) per model, plus per\-method mean±\\pm1 SD \(right\)\. C3 vectors span the full unit sphere \(cosines from−0\.99\-0\.99to\+0\.99\+0\.99\); enc\-dec vectors collapse into a narrow cone \(cosines0\.9850\.985–1\.0001\.000\) because the C1 encoder, trained on passage co\-activations, does not preserve concept\-discriminative directional signal\.
### C3 Concept Geometry per Model

Figure[10](https://arxiv.org/html/2608.05164#A10.F10)shows a 2D PCA of the 11 C3 steering vectors in each model’s native activation space\. Concepts are well\-separated across all five architectures, with semantic clusters visible: code/math and SQL concepts sit near each other, while medical and narrative fiction occupy opposite poles\. The per\-model geometry varies in orientation but preserves relative concept distances, consistent with the PCA of cluster centres \(Figure[7](https://arxiv.org/html/2608.05164#A8.F7)\) showing that the shared concept space has a stable geometric structure that survives the per\-model decoder step\.

![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/fig12_c3_pca_per_model.png)Figure 10:PCA of 11 C3 steering vectors in each model’s native activation space \(unit\-normalised, 5 panels\)\. Each point is one concept\. Orthogonal spread confirms well\-separated concept directions; semantic groupings \(code/math, legal/news, medical\) are visible in all five architectures\.
### B3\-TI Source Consistency

Figure[11](https://arxiv.org/html/2608.05164#A10.F11)reports the mean pairwise cosine between the four B3\-TI vectors aimed at the same \(target model, concept\) from four different source models\. If the MLP bridge learns a faithful and consistent translation, vectors from different sources should converge on the same target direction\. Forgpt2\-largeas target \(mean 0\.895\), source vectors agree strongly — the 0\.8B representational space imposes a tight constraint that all four bridges converge towards\. For 7B targets \(means 0\.64–0\.70\), the larger target space allows more source\-dependent variation\. The pattern mirrors the scale\-tier hierarchy in Table[2](https://arxiv.org/html/2608.05164#S4.T2): the same geometric bottleneck that limits cross\-scale alignment also makes the target space more ŕeachablef́rom any source\.

![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/fig11_ti_source_consistency.png)Figure 11:B3\-TI source consistency heatmap: mean pairwise cosine between the four B3\-TI vectors \(from four different source models\) targeted at each \(target model, concept\) cell\. High values indicate all source models agree on which direction to push the target\.gpt2\-large\(0\.8B\) shows the highest consistency \(0\.895\), consistent with its smaller, more constrained representational space\.

## Appendix KPrompt\-Sensitivity Distribution

Of the 30 evaluation prompts, 14 meet the qualifying criterion \(both B3\-TI and at least one universal method outperform same\-model native vectors on the averaged \(model, concept\) pair\)\. Table[23](https://arxiv.org/html/2608.05164#A11.T23)lists these 14 prompts in descending order of mean concept\-score delta at best positive strength\. The 4 prompts marked⋆\\starare the only ones satisfying the stricter “full hierarchy” criterion \(universal\>\>B3\-TI\>\>native\)\. Figure[16](https://arxiv.org/html/2608.05164#A11.F16)shows that steerability is bimodal: best\-quartile prompts yield consistently positiveΔ\\Deltaacross all methods, while worst\-quartile prompts produce near\-zero or negative deltas\.

Table 23:Qualifying prompts \(14/30\)\. Score = meanΔ\\Deltaacross \(model, concept\) pairs at best positive strength\.⋆\\star= full hierarchy holds\.Table[24](https://arxiv.org/html/2608.05164#A11.T24)reports mean signed delta, win rate, and delta ats=1s=1for all evaluated methods on the full 30\-prompt supervised evaluation set\.

Table 24:Full supervised results \(15 concepts, 30 prompts, 5 models\)\. \(1\) Mean signedΔ\\Deltaincluding negatives; \(2\) win rate \(% pairs withΔ\>0\\Delta\>0\); \(3\)Δ\\Deltaat fixed strengths=1s=1\.![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/fig6_prompt_sensitivity.png)Figure 12:Prompt sensitivity curves: mean concept\-scoreΔ±95%\\Delta\\pm 95\\%CI per prompt rank for each model and method class\. Prompts ranked ascending by cross\-method meanΔ\\Delta\. CI computed as normal approximation \(±1\.96⋅SE\\pm 1\.96\\cdot\\mathrm\{SE\}\)\.![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/appB_strength_curves.png)Figure 13:Strength response curves: mean concept\-scoreΔ\\Deltaas a function of injection strengthssper model and method class\. Lines show mean±\\pmSE across \(concept, prompt\) pairs\.![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/fig7_strength_boxplots.png)Figure 14:Vector strength sensitivity:Δ\\Deltadistribution per strength level\|s\|∈\{1,2,3,5\}\|s\|\\in\\\{1,2,3,5\\\}\(9 panels\)\. Box = IQR, whiskers = 5th–95th percentile, dot = mean with 95% CI, dashed line = zero\. DeepSeek\-7B at\|s\|=5\|s\|=5shows markedly inflated variance\.![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/fig8_best_worst_prompts.png)Figure 15:Best vs worst performing prompts: meanΔ±95%\\Delta\\pm 95\\%CI per method class \(top/bottom quartile prompts ranked by cross\-method meanΔ\\Delta\)\.▲\\blacktriangle= best quartile,▼\\blacktriangledown= worst quartile\.![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/appF_prompt_best_worst_detail.png)Figure 16:Best vs worst prompts per model and method \(5 sub\-panels\)\. Solid bars: mean concept\-scoreΔ\\Deltafor best\-quartile prompts \(top 25% ranked by cross\-method meanΔ\\Delta\); hatched bars: worst\-quartile prompts\. All five methods shown per model\. The gap between best and worst quartiles is largest formistral\-7bandgemma\-2\-2band smallest fordeepseek\-llm\-7b\(restricted tos=1s=1, compressing the dynamic range\)\. This confirms that prompt register interacts strongly with steering efficacy, and that register\-neutral prompts are necessary for fair cross\-method comparison\.
## Appendix LFull Functional Results, Weak Domains

Table[24](https://arxiv.org/html/2608.05164#A11.T24)\(above, in Appendix[K](https://arxiv.org/html/2608.05164#A11)\) provides mean delta, win rate, ands=1s=1delta for all methods and all five models including the nine low\-pass\-rate domains\. The narrative below summarises the pattern for those nine domains\.

The nine domains with B2 pass rate below 50% —creative\_writing,legal,code\_python,code\_sql,math\_reasoning,code\_snippets,sentiment,science\_biomedical,question\_answering— show mixed or negative Cohen’sddvalues in B2 \(Table[4](https://arxiv.org/html/2608.05164#S4.T4)\), indicating that the SAE features selected for those domains are not consistently discriminative at the corpus level\. For these domains, all methods \(native SAE, B3\-TI, Univ\) produce mixed signed deltas across models: no method is systematically above 50% win rate when averaged over the full 30\-prompt set\. This is mechanistically expected: concept discriminability in feature space \(Cohen’sdd\) is a precondition for steerability\. Whered<0\.4d<0\.4, the steering vector direction is noisy and its injection does not reliably shift output distribution toward the target register\. The distinction between the six strong domains \(Table[4](https://arxiv.org/html/2608.05164#S4.T4), bold\) and the nine weak domains is therefore grounded in a measurable upstream property of the feature representation, not a post\-hoc selection on evaluation outcome\. All results are included in the released evaluation files \(full\_eval\_results\.jsonl\) for completeness and reproducibility\.

## Appendix MFuture Work

The highest\-priority deferred experiments are the preregistereddisjoint\-corpus ablation\(independent corpus subsets per model, to rule out shared\-statistics as the driver of alignment\) andT3 activation\-patching causal tracing\(inserting a steering vector at one layer and patching the residual stream at subsequent layers to trace the causal path, which would upgrade the T2 bidirectionality evidence to mechanistic circuit\-level attribution\)\.

Multi\-layer injection\.All experiments use single\-layer injection at≈\\approx50% depth for clean comparison with prior work\. Sweeping injection depth and aggregating across layers may reduce prompt\-sensitivity and improve transfer to GPT\-2\-large\.

Representation\-side questions\.It remains open whether SAE architecture choices \(TopK vs\. ReLU, expansion factor,kkschedule\) meaningfully affect the cross\-model alignment signal, and whether the 15\-concept supervised label set is a bottleneck for feature\-pair recall\.

Model family extensions\.Mamba, mixture\-of\-experts, and encoder\-decoder architectures are priority extensions \(the C3\-EncDec cone collapse is a candidate mechanism to study in cross\-architecture crosscoders\[Lindseyet al\.,[2024](https://arxiv.org/html/2608.05164#bib.bib2)\]\); non\-English and multilingual models are also out of scope for the current study\.

Closed\-source models\.All experiments require mid\-layer activation access; closed\-source frontier systems \(GPT\-4, Claude, Gemini\) lie outside current scope\. Developing output\-space or black\-box analogues of the alignment pipeline—via paired\-output representation probing or distillation into an open proxy—is a necessary step before these results can generalise to closed\-source deployment contexts\.

## Appendix NLLM\-Judge Validation

As a qualitative complement to DeBERTa concept\-score deltas, a sample of 650 steered outputs was submitted to Claude \(claude\-haiku\-4\-5, temperature 0\) with a forced\-choice prompt: given the original unsteered output and the steered output, the judge rates whether the steered output more strongly expresses the target concept \(win\) or not\. Table[25](https://arxiv.org/html/2608.05164#A14.T25)reports Claude judge win rates on the 14 qualifying prompts at two prompt\-set sizes\.

Table 25:Claude judge win rate \(% steered output rated as expressing target concept more strongly\) on qualifying prompts \(⋆\\star= full hierarchy prompts only; “Top\-14” = all qualifying prompts\)\. Only cells with≥\\geq3 valid comparisons are shown; others are marked —\. Note: the LLM judge is a holistic quality probe; it is not used to confirm statistical significance\.![Refer to caption](https://arxiv.org/html/2608.05164v1/figures/fig5_judge_win_rates.png)Figure 17:LLM judge \(claude\-haiku\-4\-5\) win rates: steered output vs baseline\. Blue bars indicate win rate≥\\geq50%; red bars below chance\. Sorted by win rate descending\.The LLM judge and the DeBERTa concept\-score delta are independent probes\. DeBERTa measures the shift in the token\-level distribution over concept labels; the LLM judge measures holistic output quality relative to the target register\. On the qualifying prompts, Univ C3 achieves the highest LLM\-judge win rate \(38\.0–60\.0% depending on prompt subset\), with deepseek showing 100% on the top\-5 prompts despite its low DeBERTa win rate, indicating that when deepseek does produce on\-topic output it is qualitatively strong\. The Pearson correlation between DeBERTaΔ\\Deltaand LLM\-judge outcome isr=−0\.004r=\-0\.004\(n=228n=228\), confirming the two metrics measure distinct aspects of the steering effect and should not be treated as interchangeable\. All Claude judge prompts and raw outputs are stored inresults/llm\_judge\_results\_prod\.jsonl\.

## Appendix ODiscarded Approaches

#### Run 1: Hungarian pair extraction\.

An earlier run usedscipy\.linalg\.linear\_sum\_assignment\(Hungarian method\) for pair extraction, yielding 1,823 pairs with a B2 pass rate of 68\.4%\. This inflated result arose from three sources: \(i\) Hungarian forces global 1:1 coverage including weak pairs, but pre\-selection of the top\-KKfeatures by SAE label confidence guaranteed high discriminativity before B2 ran; \(ii\) the Cohen’sddmean was inflated from 0\.423 \(Run 2\) to 1\.213 \(Run 1\) by this pre\-selection; \(iii\) gpt2\-large cross\-scale pairs appeared to pass at 65–79% due to undirected pooling that masked the scale gap\. All main\-paper results use Run 2 \(MNN extraction, 3,308 directed pairs, 45\.5% overall pass rate\)\. The 45\.5% is not a regression; it is a more honest estimate that correctly separates three scale\-tier populations\. The correct comparison is Run 2 7B↔\\leftrightarrow7B \(48\.9%\) vs Run 1 same\-scale pairs \(43–54% after bias correction\)—consistent\.

#### HDBSCAN without UMAP \(C2 Runs 1–2\)\.

Two early C2 runs applied HDBSCAN directly to the 512\-d shared concept space\. Run 1 \(min\_cluster\_size=20\): 58% noise\. Run 2 \(min\_cluster\_size=50\): 65% noise \(worse, due to over\-merging\)\. Root cause: the curse of dimensionality causes pairwise distances in 512\-d cosine space to concentrate near their mean, eliminating density gradients required by HDBSCAN\[Beyeret al\.,[1999](https://arxiv.org/html/2608.05164#bib.bib27)\]\. Fix: UMAP reduction to 30d reduced noise to 15\.8% \(Run 3, reported in main paper\)\.

#### B3 SAE\-Feature Locator and B3 Guided CAA \(insufficient feature coverage\)\.

Two B3 strategies that operate directly on the B2\-validated pair list degenerated and are excluded from the main evaluation\.

The*B3 SAE\-Feature Locator*constructs a target steering vector by retrieving the target\-model SAE feature that is B2\-validated as corresponding to each of the guide’s top\-3 concept features, then accumulating confidence\-weighted target decoder columns\. The method fails because B2 yields only 12–85 validated pairs per directed model pair and none of these contain any of the top\-3 concept features for any of the 15 evaluated concepts\. The locator therefore falls back to writing the guide’s own decoder weights as the output, producingsae\_decoder\_vectorentries numerically identical to the native A5 SAE\-decoder vector for all 20 pairs×\\times15 concepts = 300 evaluation cells\. The root cause is architectural: B2\-passing pairs are MNN\-selected for corpus\-level cosine alignment, not for concept\-label membership; the probability that any specific concept’s top\-concept features fall in the MNN\-matched set is substantially lower than the 45\.5% aggregate pass rate suggests\.

The*B3 Guided CAA*produces a mean\-difference vector from the target model’s residual streams, split using the*guide*’s concept activity \(above vs\. below median on that concept’s top features\) rather than corpus labels\. A code defect prior to 2 May 2026 caused this path to duplicate the native CAA vector; after correctioncaa\_cross\_vectoris genuinely distinct, but achieves no consistent advantage over native vectors in preliminary evaluation at the tested strength range\. Both B3 BAL vectors are retained in the release undercross\_model\_steering\_vectors\_bal\.jsonas an audit trail but are not included in any reported comparison\.

## Appendix PCompute Budget

Table 26:Estimated A100\-80GB GPU hours per experiment step\.
## Appendix QReleased Assets

Table[27](https://arxiv.org/html/2608.05164#A17.T27)summarises the released artefacts, their locations within the dataset repository, and applicable licences\.

Table 27:Released artefacts\. All are hosted atayushi\-agarwal/universal\-steeringon Hugging Face unless marked\(git\)\. Licence applies to the artefact itself; use is additionally constrained by the upstream model licence where noted\.Reproduction instructions are provided asREADME\.mdin the code repository with exact commands and expected outputs for each pipeline step\. All randomness usesseed=42throughout\. Large binary assets are downloaded automatically by runningpython pipeline/a1\_download\_data\.py\(requires a Hugging Face token and acceptance of per\-model terms\)\.

## Appendix RAsset Licences

Model licences: GPT\-2 \(MIT\), Mistral\-7B\-v0\.3 \(Apache 2\.0\), LLaMA\-3\.1 \(LLaMA 3 Community Licence, permits research and commercial use below 700M MAU\), Gemma\-2\-2B \(Gemma Terms of Use, permits research\), DeepSeek\-LLM\-7B \(DeepSeek Licence, permits research\)\. Dataset licences range from MIT \(GSM8K, MetaMathQA\) to CC\-BY\-SA\-3\.0 \(Yelp, Wikipedia\); the derived corpus is released under CC\-BY\-SA\-4\.0\.

## Appendix SUse of Large Language Models

Claude was used in three distinct roles in this pipeline, using different model versions matched to each task’s complexity and cost requirements\.

Feature labelling \(A4\) — Claudeclaude\-sonnet\-4\-5:The top\-10 activating corpus passages per SAE feature were submitted toclaude\-sonnet\-4\-5\(temperature 0\) with a prompt requesting a 2–4\-word domain label insnake\_caseformat; no few\-shot examples were included to avoid priming toward the 15 training domains\. Sonnet was selected here because label quality is directly upstream of B1/B2 alignment statistics\.

Universal concept naming \(C2\) — Claudeclaude\-sonnet\-4\-5:The top\-5 activating passages per HDBSCAN cluster were submitted toclaude\-sonnet\-4\-5\(temperature 0\) with an open\-ended naming prompt; out\-of\-corpus compound\-domain examples \(e\.g\.,sql\_and\_legal\) were included to encourage cross\-domain labels and prevent single\-domain priming\.

LLM judge \(D1 evaluation\) — Claudeclaude\-haiku\-4\-5:650 steered outputs were evaluated byclaude\-haiku\-4\-5\(temperature 0\) using a forced\-choice prompt: given the original unsteered output and the steered output, the judge rated which more strongly expresses the target concept\. Haiku was selected for the judge role because of the large evaluation volume \(650 comparisons\) and because judge precision requirements are lower than labelling precision — the holistic quality comparison is binary and does not require fine\-grained semantic reasoning\.

All prompts and raw Claude outputs are stored verbatim in the released feature label JSON files \(one file per model\), the universal concept JSON file, andresults/llm\_judge\_results\_prod\.jsonlrespectively\. No LLM output was used in threshold selection, statistical inference, or result interpretation; all such decisions were made by the authors\. Claude \(claude\-sonnet\-4\-5\) was additionally used for grammar revision of the manuscript\.