NinaXander: Feasibility and Limits of Composing Frozen Language Models Across Architecture Families via a Shared Latent Space

# NinaXander: Feasibility and Limits of Composing Frozen Language Models Across Architecture Families via a Shared Latent Space
Source: [https://arxiv.org/html/2609.38261](https://arxiv.org/html/2609.38261)
Takanori Kotama and Shun\-ichiro HayashiDaichi Mukunoki, Tetsuya Hoshino and Takahiro KatagiriAffiliation:Information Technology Center Nagoya University

###### Abstract

In this paper we propose NinaXander, a series of composed language models obtained by connecting layers of frozen language models from different architecture families with a single trained shared\-latent adapter\. A composed model runs the first layers of one model, converts the resulting intermediate representation once with the adapter, and then runs the remaining layers of the other model\. Once the adapter is trained, several composed models that connect at different layers are obtained without retraining\. Using the recurrent RWKV\-4\-Raven\-7B and the Transformer\-based Tulu\-Pythia\-6\.9b, abbreviated as RWKV and Pythia, this study examines whether frozen models from different families can be recombined post hoc\. The composed models answered multiple\-choice questions, and those whose generations we examined produced syntactically well\-formed text\. The configuration that combines the first55layers of Pythia with the remaining2727layers of RWKV reduced the Transformer key\-value \(KV\) cache by84\.4%84\.4\\%with accuracy not significantly different from that of RWKV alone\. In multiple\-choice accuracy, however, no composed model matched the parent model Pythia, and language\-modeling performance decreased sharply on WikiText, a corpus of Wikipedia articles outside the training domain\. The correspondence between intermediate representations was also obtained in one favorable case, with a shared tokenizer, the same depth, and the same hidden width, and does not show that the models share a general semantic space\.

## IIntroduction

Pretraining a language model requires an enormous amount of computation, yet its result remains only as weights fixed to a particular architecture\. The Transformer, which most current language models use, keeps the keys and values of past tokens as a key\-value \(KV\) cache to compute self\-attention, and the size of this cache grows in proportion to the context length\. Existing ways of changing inference\-time characteristics through the architecture, for example to reduce the KV cache, either pretrain a new hybrid model that incorporates recurrent layers or replace some layers of an existing model with a lighter mechanism and retrain it\. Both approaches retrain weights; neither recombines trained blocks as frozen components\.

In this paper, a family is a group of models that share the same kind of architecture, such as Transformers or RWKV \(Receptance Weighted Key Value\), which computes linear attention in a recurrent form and so runs in time linear in the sequence length\. Studies that connect frozen language models to one another have been reported, but in these studies either both connected models are Transformers or the whole composed model is fine\-tuned, and recombining trained checkpoints across families as frozen components has not yet been examined\.

This study examines whether the intermediate representations of frozen models from different families can be converted into each other by a small trained network, called the adapter below, so that the first layers of one model can be connected to the remaining layers of another\. We call a model composed in this way a chimera model and the original models used for the composition parent models, and we name the series of chimera models built with a single adapter NinaXander111The name NinaXander comes from a chimera in Hiromu Arakawa’s manga*Fullmetal Alchemist*\. The composition works but falls short of what it was made from, which corresponds to the results of this study\.\. Once the adapter is trained, several chimera models that connect at different layers are obtained without retraining\. The parent models are RWKV\-4\-Raven\-7B, a version of RWKV further trained to follow instructions, and the Transformer\-based Tulu\-Pythia\-6\.9b\. The two models satisfy the favorable conditions of sharing a tokenizer, which splits text into token sequences, and of having the same hidden width and the same number of layers\. Below, the two models are abbreviated as RWKV and Pythia, respectively\.

The main purpose of this study is not to build a practical high\-performance model but to clarify whether frozen models from different families can be connected by converting their intermediate representations\. In our experiments the chimera models answered multiple\-choice questions, and those whose generations we examined produced syntactically well\-formed text\. We also obtained a configuration that reduces the Transformer layers to55, decreasing the KV cache size by84\.4%84\.4\\%, with accuracy not significantly different from that of RWKV alone\. On the other hand, no configuration exceeded the stronger parent model, and RWKV alone, which has no KV cache, was significantly exceeded in accuracy by only one chimera model, on the science question set SciQ\. The contributions of this study are the construction and evaluation of chimera models that connect frozen language models from different families in both directions with a single adapter; an analysis of the factors that limit the performance of the connection, through a comparison with reconstruction within the same model and an intervention inside the adapter; and measurements of accuracy, KV cache size, and performance on text from domains not used to train the adapter\.

## IIBackground and Related Work

### II\-AResidual Stream and Inference Cost

An autoregressive language model updates the representation at each token position, called the residual stream, layer by layer\. Blockℓ\\ell, counted from00, of a model with hidden widthddreceives the inputhℓ−1h^\{\\ell\-1\}and outputshℓ=hℓ−1\+fℓ​\(hℓ−1\)h^\{\\ell\}=h^\{\\ell\-1\}\+f\_\{\\ell\}\(h^\{\\ell\-1\}\), whereh−1h^\{\-1\}is the token embedding\. The functionfℓf\_\{\\ell\}performs sequence mixing, which mixes information across tokens, and channel mixing, which transforms features within each token position\. Even when two models have the same hidden width and depth, thehℓh^\{\\ell\}of one model lies in a coordinate system determined by that model’s training, and passing it without conversion to blockℓ\+1\\ell\{\+\}1of the other model does not match the distribution that the receiving weights expect\. Whether two models can be connected at an intermediate layer therefore reduces to whether the residuals of the two families can be converted into each other\.

The two families also differ in how their inference cost grows\. The self\-attention of Transformers weights tokens by their similarity normalized with the softmax function, and because it keeps the keys and values of all past tokens it needs a cache of sizeO⁡\(T\)O\(T\)for a context lengthTT\. A layer that can run as linear attention or in an equivalent recurrent form keeps only a fixed\-size state, which isO⁡\(1\)O\(1\)\. Which of the two connected models is placed on the input side changes how many Transformer layers remain and how much cache must be kept\.

### II\-BModel Stitching

Lenc and Vedaldi\[[1](https://arxiv.org/html/2609.38261#bib.bib1)\]introduced connecting layers of22frozen models through a trainable stitching layer as a way to measure the equivalence of representations\. For two convolutional neural networks \(CNNs\), they built a “Franken\-CNN” that connects the lower layers of one network to the upper layers of the other through a linear stitching layer, and evaluated it with the loss of the original task\. The pairs they studied include networks trained on different data, namely the large\-scale image classification dataset ImageNet and the scene recognition dataset MIT Places built by the Massachusetts Institute of Technology \(MIT\)\. They also include pairs of CNNs with different architectures: the 8\-layer AlexNet, the 16\-layer VGG\-16 proposed by the Visual Geometry Group of the University of Oxford, and the 50\-layer ResNet\-50 with residual connections\. Bansal et al\.\[[2](https://arxiv.org/html/2609.38261#bib.bib2)\]revisited stitching as a means of comparing representations\. They stitched models of the same architecture that differ in random seed, amount of training, and width, and also stitched across different objectives, namely supervised learning and self\-supervised learning, which trains on pseudo\-labels constructed from the data itself\. Stitching across different architectures, however, remained an open problem\. Csiszárik et al\.\[[3](https://arxiv.org/html/2609.38261#bib.bib28)\]matched the intermediate representations of two trained networks with a single affine stitching layer and proposed a framework that evaluates their agreement not only by similarity in representation space but also functionally, by the task performance of the stitched network\. They further showed that existing measures of representational similarity do not necessarily reflect task performance\. Following this position, we evaluate separately the degree to which representations agree, called alignment below, and the degree to which one representation can be recovered from the other, called decodability below and measured in this paper by the read\-out accuracy \(§[III\-E](https://arxiv.org/html/2609.38261#S3.SS5)\)\. All of these studies target image recognition models\. This study differs in that it connects two different families, recurrent neural networks \(RNNs\) and Transformers, through a common latent space in which the representation is converted once at an intermediate layer\.

### II\-CComposing Frozen Large Language Models

Recent studies compose frozen large language models \(LLMs\) through trainable connection layers\. StitchLLM\[[4](https://arxiv.org/html/2609.38261#bib.bib12)\]connects blocks of Transformers of different sizes through stitching layers and uses a lightweight routing mechanism to decide how much computation to allocate to each query\. BTS \(Branch\-Train\-Stitch\)\[[5](https://arxiv.org/html/2609.38261#bib.bib13)\]places several frozen expert models, derived from the same seed model for specific domains, alongside the frozen seed model, which serves as the hub, and inserts Experts\-into\-Hub and Hub\-into\-Experts stitching layers alternately every few layers to exchange hidden states in both directions\. CALM \(Composition to Augment Language Models\)\[[6](https://arxiv.org/html/2609.38261#bib.bib14)\]fuses the intermediate representations of two frozen Transformers through cross\-attention, which uses the representation of one model as queries to attend to the representation of the other\. Manticore\[[7](https://arxiv.org/html/2609.38261#bib.bib15)\]runs block sequences of a Transformer and of Mamba, a selective state\-space model, in parallel with projections, and mixes their outputs with learned weights that are non\-negative and sum to 1\. This hybrid model is fine\-tuned as a whole, and its embedding layer and its language\-model head \(LM head\), which outputs probabilities over the vocabulary, are newly initialized\.

Other studies convert activations directly between frozen models\. Moschella et al\.\[[8](https://arxiv.org/html/2609.38261#bib.bib29)\]showed that relative representations, which re\-express each representation by its similarity to a small set of reference samples called anchors, remove the effect of the transformation between independently trained latent spaces, so that models can be stitched without training a converter\. Maiorca et al\.\[[9](https://arxiv.org/html/2609.38261#bib.bib30)\]estimate a map between two latent spaces in closed form from a small number of semantic correspondences\. As low\-capacity direct maps corresponding to these methods, this study uses affine maps for comparison \(§[IV\-C](https://arxiv.org/html/2609.38261#S4.SS3)\)\. Because our affine maps are fitted directly with corresponding residuals as targets, the comparison is more favorable to them than to relative representations, which do not use paired data for training\.

Model Alignment Search\[[10](https://arxiv.org/html/2609.38261#bib.bib16)\]maps activations in both directions between a frozen recurrent network and a frozen Transformer, but it is a method for measuring representational similarity and does not build a composed model\. Chen et al\.\[[11](https://arxiv.org/html/2609.38261#bib.bib17)\]map the residual streams of two frozen language models in both directions with a single affine layer, and evaluate, down to the cross\-entropy computed from next\-token prediction probabilities, composed models that connect the first layers of one model to the remaining layers of the other\. Their pairs, however, are Transformers of one model series sharing a tokenizer; the future work they name is validation across model series, and they do not discuss different families\.

The Bicameral Model\[[12](https://arxiv.org/html/2609.38261#bib.bib18)\], a preprint, couples two frozen language models through bidirectional conversion networks, but each model runs in parallel while executing all of its own layers\. GoldFinch\[[13](https://arxiv.org/html/2609.38261#bib.bib19)\], also a preprint, stacks a Transformer on a structure that extends RWKV\-6 \(Finch\), an improved version of RWKV\-4, and compresses the KV cache by756756–25502550times, but it is trained from an initial state rather than combining pretrained parent models\. Oozeer et al\.\[[14](https://arxiv.org/html/2609.38261#bib.bib34)\]learn activation maps between frozen LLMs and transfer steering vectors that control tendencies of the output, mainly between smaller and larger models of one series, using them to remove backdoors planted so that specific inputs trigger undesired outputs and to disable the refusal of harmful instructions\. To validate a map, they replace the activations of one target layer with mapped ones and generate text, and they use affine maps as a control, so their work is close to ours; each map, however, serves one layer pair, and both models are Transformers\.

### II\-DRepresentation Similarity

Standard measures of representational similarity between networks are centered kernel alignment \(CKA\)\[[15](https://arxiv.org/html/2609.38261#bib.bib3)\]and two extensions of canonical correlation analysis \(CCA\), singular vector CCA \(SVCCA\) and projection weighted CCA \(PWCCA\)\[[16](https://arxiv.org/html/2609.38261#bib.bib4),[17](https://arxiv.org/html/2609.38261#bib.bib5)\]\. Of these, we use linear CKA, together with the fittedR2R^\{2\}of per\-layer linear maps, as auxiliary indicators to examine the correspondence across all layers, and focus on functional stitching through reconstruction\. On the universality of representations across families, Wang et al\.\[[18](https://arxiv.org/html/2609.38261#bib.bib35)\]compared Transformers and Mamba\. They extracted the features of both models with sparse autoencoders, which are trained to express the input as a combination of a small number of features, and reported that most features are similar\. They further showed that the structure of induction circuits, which predict the next token by referring to repeated patterns in the context, is also similar\. Their subject, however, is Mamba, not the RWKV family\.

### II\-EModel Merging and Knowledge Fusion

Methods that merge models in weight space include model soups\[[19](https://arxiv.org/html/2609.38261#bib.bib6)\], which average the weights of several models fine\-tuned from the same pretrained model; TIES\-Merging\[[20](https://arxiv.org/html/2609.38261#bib.bib7)\], which merges task\-specific weight differences after resolving sign conflicts; and evolutionary model merge\[[21](https://arxiv.org/html/2609.38261#bib.bib8)\], which searches for merging recipes by evolutionary computation\. These methods assume identical or compatible weights\. FuseLLM\[[22](https://arxiv.org/html/2609.38261#bib.bib9)\], which fuses the output distributions of several models, does not need compatible weights, but it needs a procedure that aligns tokens between the vocabularies of different tokenizers and additional training of the target model\. This study merges no weights and recombines frozen blocks by converting in activation space, which makes it complementary to these methods\.

### II\-FPosition of This Study

Among the studies above, those that compose language models either connect two Transformers, fine\-tune the composed model as in Manticore, or use at most one pretrained parent model\. A stitching scheme with a single connection point, connections in both directions, and frozen parent models has already been demonstrated by Chen et al\.\[[11](https://arxiv.org/html/2609.38261#bib.bib17)\]for pairs of Transformers within one model series that share a tokenizer, and this study does not claim the scheme itself as a new contribution\. What this study addresses is the case of different families, which Chen et al\. did not examine: we test whether the same stitching holds between a recurrent model based on linear attention and a Transformer\. To our knowledge, no study has composed these two families by connecting them at a single point with both parent models and the LM heads frozen and evaluated the result as a generator\. We also report how much the composition reduces the Transformer KV cache\.

## IIIMethod

### III\-ATarget Models

We use RWKV\-4\-Raven\-7B and Tulu\-Pythia\-6\.9b, instruction\-following models tuned to respond according to user instructions222The experiment code, raw evaluation data, and reproduction procedures are available at[https://github\.com/Katagiri\-Hoshino\-Lab/NinaXander](https://github.com/Katagiri-Hoshino-Lab/NinaXander)\.\. Below, the former is denoted modelAAand the latter modelBB\. These symbols are used in the notation for residuals in §[III\-B](https://arxiv.org/html/2609.38261#S3.SS2)and for read\-outs in §[III\-D](https://arxiv.org/html/2609.38261#S3.SS4)\. Both models have3232blocks and a hidden width ofd=4096d\{=\}4096, and they share the tokenizer of GPT\-NeoX\-20B, a language model developed by the non\-profit research group EleutherAI\. The same input is therefore converted into the same token sequence, and the residuals at each position can be paired directly\. Because the depths are equal, thejjth blocks of the two models can be paired, and because the hidden widths are equal, the vectors at the connected layer have the same dimension\. These three conditions were chosen deliberately to make the connection easier, and pairs of models whose tokenizers, depths, or hidden widths differ are outside the scope of this study\.

The base models of both were pretrained on The Pile\[[23](https://arxiv.org/html/2609.38261#bib.bib21)\], an825825GiB English corpus that EleutherAI built from2222kinds of data, including academic papers, web documents, books, and source code\. Although the base corpus is shared, the composition of the data and the prompt formats used for additional training on pairs of instructions and responses, called instruction tuning below, differ between the two models\.

The RWKV architecture\[[24](https://arxiv.org/html/2609.38261#bib.bib10)\]replaces softmax dot\-product attention with linear\-time sequence mixing by the WKV operator, which averages the values of the tokens up to the current one with weights set by their keys and decayed over time, and it can run the same computation either in parallel form or in recurrent form\. The Raven model used here is an instruction\-following version of RWKV\-4 developed by the RWKV project and released by its developer BlinkDL\. RWKV\-4\-Pile\-7B, pretrained on The Pile itself, is further trained on data that include Alpaca\[[25](https://arxiv.org/html/2609.38261#bib.bib24)\], the instruction\-response data released by Stanford University, the instruction data Guanaco and GPT4All, Code\-Alpaca, and ShareGPT\. Code\-Alpaca is synthetic data of about20,00020\{,\}000instructions and responses centered on code generation, editing, and optimization\[[26](https://arxiv.org/html/2609.38261#bib.bib22)\]\. ShareGPT collects multi\-turn conversations with ChatGPT, the dialogue service of OpenAI, shared by its users\[[27](https://arxiv.org/html/2609.38261#bib.bib23)\]\. We convert the RWKV\-4\-Raven\-7B released by BlinkDL into the format used on the model\-sharing platform Hugging Face and re\-save it in 16\-bit floating point \(FP16\)333BlinkDL, RWKV\-4 Raven model repository,[https://huggingface\.co/BlinkDL/rwkv\-4\-raven](https://huggingface.co/BlinkDL/rwkv-4-raven)\.\. The weights we use have7,392,649,2167\{,\}392\{,\}649\{,\}216parameters,3232blocks, a hidden width of40964096, and an intermediate width of1638416384in channel mixing\. The converted configuration setscontext\_lengthto10241024\. This value is defined as the maximum sequence length handled in a single forward pass, and in recurrent form the model can propagate a state whose size does not depend on the context length444Hugging Face Transformers, RWKV documentation,[https://huggingface\.co/docs/transformers/model\_doc/rwkv](https://huggingface.co/docs/transformers/model_doc/rwkv)\.\. The file name and revision of the source weight file, however, were not recorded, so the context length used to train the source checkpoint cannot be identified\. Being recurrent also does not guarantee language performance on sequences longer than the training or configured length\.

The Pythia series\[[28](https://arxiv.org/html/2609.38261#bib.bib11)\]consists of Transformer models that EleutherAI developed for research and trained with its GPT\-NeoX library\. Tulu\-Pythia\-6\.9b is Pythia\-6\.9b with all of its parameters fine\-tuned by the Allen Institute for AI \(AI2\), a non\-profit research institute, on the mixed instruction data Tulu\[[29](https://arxiv.org/html/2609.38261#bib.bib20)\]\. The mixture contains FLAN V2, which converts many natural language processing tasks into instruction format and whose name stands for Finetuned Language Net; Chain\-of\-Thought, which collects answers that include reasoning steps; Dolly, instruction data written by employees of Databricks; OpenAssistant 1, conversation data collected by crowdsourcing; GPT4\-Alpaca, in which GPT\-4, a large language model of OpenAI, responds to the Alpaca instructions; and Code\-Alpaca and ShareGPT555Allen Institute for AI, open\-instruct\-pythia\-6\.9b\-tulu model card,[https://huggingface\.co/allenai/open\-instruct\-pythia\-6\.9b\-tulu](https://huggingface.co/allenai/open-instruct-pythia-6.9b-tulu)\.\. The weights we use have6,856,040,4486\{,\}856\{,\}040\{,\}448parameters,3232blocks, a hidden width of40964096, and3232attention heads\. The maximum number of positions in the configuration is20482048666The released configuration file is[https://huggingface\.co/allenai/open\-instruct\-pythia\-6\.9b\-tulu/blob/main/config\.json](https://huggingface.co/allenai/open-instruct-pythia-6.9b-tulu/blob/main/config.json), which setsmax\_position\_embeddingsto20482048\.\. According to their publishers, the instruction\-tuning data of both models include datasets named Code\-Alpaca and ShareGPT, and Tulu’s GPT4\-Alpaca uses the same instructions as Raven’s Alpaca\. Raven’s publisher does not state versions, so the extent of the overlap cannot be determined, but the instruction data of the two models are not independent\. In this paper, block and layer are used as synonyms\.

### III\-BResiduals and Notation

In this paper a residual does not mean an error; it is the hidden vector passed from each block to the next\. Connecting two models means converting this vector from the coordinate system of one model into that of the other and passing it on\. We writehMj​\[t\]∈ℝ4096h\_\{M\}^\{j\}\[t\]\\in\\mathbb\{R\}^\{4096\}for the unstandardized residual that blockjjof modelM∈\{A,B\}M\\in\\\{A,B\\\}outputs at token positiontt\. It corresponds tohℓh^\{\\ell\}in §[II](https://arxiv.org/html/2609.38261#S2), and the block index isj∈\{0,…,31\}j\\in\\\{0,\\ldots,31\\\}\. We feed the same token sequence to both models and takehAj​\[t\]h\_\{A\}^\{j\}\[t\]andhBj​\[t\]h\_\{B\}^\{j\}\[t\]as one training example\.

The scale of the residuals differs between models and layers, so we compute the mean and standard deviation of each dimension from the training data of each layer\.stdMj\\mathrm\{std\}\_\{M\}^\{j\}standardizes a residual with these statistics, andunstdMj\\mathrm\{unstd\}\_\{M\}^\{j\}is the inverse operation that restores the original scale\. The standardized residual is writtenxMj​\[t\]=stdMj​\(hMj​\[t\]\)x\_\{M\}^\{j\}\[t\]=\\mathrm\{std\}\_\{M\}^\{j\}\(h\_\{M\}^\{j\}\[t\]\)\. In the training objective below the layer indexjjand the token positionttare omitted for brevity, and we writexA,xBx\_\{A\},x\_\{B\}\.

### III\-CShared\-Latent Adapter

Our adapter maps the intermediate representations of the two parent models into a common vector space, called the shared latent space below, and recovers the intermediate representation of the other model from this space\. We call this adapter the shared\-latent adapter\. It has one encoder and one decoder for each model\.EA,EBE\_\{A\},E\_\{B\}are encoders that map standardized residuals into the shared latent space, andDA,DBD\_\{A\},D\_\{B\}are decoders that recover each model’s standardized residuals from the shared latent space\. The subscriptsA,BA,Bindicate the model to which each map belongs\.EA,EB,DA,DBE\_\{A\},E\_\{B\},D\_\{A\},D\_\{B\}form a single adapter shared by all3232layers, with no conditioning on the layer index\. Differences between layers are handled only by the statistics of standardization and unstandardization\.zA,zB∈ℝdzz\_\{A\},z\_\{B\}\\in\\mathbb\{R\}^\{d\_\{z\}\}are the two latent vectors obtained at the same token position, and the latent width isdz=4096d\_\{z\}\{=\}4096\. We call recovering a residual from the shared latent space a read\-out\. A read\-out that uses the encoder and decoder of the same model is a self\-reconstruction, and a read\-out that combines the encoder of one model with the decoder of the other is a cross\-family read\-out\. Withλ≥0\\lambda\\geq 0the hyperparameter that weights the latent alignment error against the two self\-reconstruction errors, the training objective is defined as follows\.

zA\\displaystyle z\_\{A\}=LN\(EA\(xA\)\),zB=LN\(EB\(xB\)\),\\displaystyle=\\mathrm\{LN\}\(E\_\{A\}\(x\_\{A\}\)\),\\qquad z\_\{B\}=\\mathrm\{LN\}\(E\_\{B\}\(x\_\{B\}\)\),\(1\)ℒ\\displaystyle\\mathcal\{L\}=‖DA​\(zA\+ξA\)−xA‖2\+‖DB​\(zB\+ξB\)−xB‖2\\displaystyle=\\\|D\_\{A\}\(z\_\{A\}\{\+\}\\xi\_\{A\}\)\{\-\}x\_\{A\}\\\|^\{2\}\+\\\|D\_\{B\}\(z\_\{B\}\{\+\}\\xi\_\{B\}\)\{\-\}x\_\{B\}\\\|^\{2\}\+λ​‖zA−zB‖2\.\\displaystyle\+\\lambda\\\|z\_\{A\}\{\-\}z\_\{B\}\\\|^\{2\}\.The first and second terms of Eq\. \([1](https://arxiv.org/html/2609.38261#S3.E1)\) are the self\-reconstruction errors in restoring the residuals ofAAandBBin the spaces of their own models\. The third term is the latent alignment error, which pulls togetherzAz\_\{A\}andzBz\_\{B\}obtained at the same token position\. The largerλ\\lambdais, the larger the relative weight of the third term in the training lossℒ\\mathcal\{L\}\. Each squared norm is computed as a mean squared error over all tokens and dimensions\.LN\\mathrm\{LN\}is a non\-affine layer normalization with no learnable scale or bias\.ξA,ξB∼𝒩⁡\(0,σ2​I\)\\xi\_\{A\},\\xi\_\{B\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\)are independent Gaussian noise terms with mean00added just before decoding\.σ\>0\\sigma\>0is the hyperparameter that sets the standard deviation of the noise, andIIis thedz×dzd\_\{z\}\{\\times\}d\_\{z\}identity matrix\. In training the adapter we fixλ=1\\lambda\{=\}1andσ=0\.4\\sigma\{=\}0\.4\. Settingλ=1\\lambda\{=\}1gives the same coefficient11to the three mean\-squared\-error terms and does not mean that the terms take equal values during training\. We did not tune these values per layer, direction, or task, nor run ablation experiments that vary them\.

The loss contains no term that predicts the residual ofBBdirectly from that ofAA, nor one that predicts the residual ofAAfrom that ofBB\. In a cross\-family read\-out,zAz\_\{A\}obtained with the encoder ofAAis fed to the decoder ofBBto giveDB​\(zA\)D\_\{B\}\(z\_\{A\}\), and the read\-out fromBBtoAAusesDA​\(zB\)D\_\{A\}\(z\_\{B\}\)\. For these maps to work, the third term must bringzAz\_\{A\}andzBz\_\{B\}into the same coordinate system\. Because the method uses paired residuals, we do not call it unsupervised learning, even though no cross\-family target is given directly\.

xAx\_\{A\}EAE\_\{A\}zAz\_\{A\}DAD\_\{A\}x^A\\hat\{x\}\_\{A\}xBx\_\{B\}EBE\_\{B\}zBz\_\{B\}DBD\_\{B\}x^B\\hat\{x\}\_\{B\}\+ξA\+\\xi\_\{A\}\+ξB\+\\xi\_\{B\}LNLNA→AA\\\!\\to\\\!AB→BB\\\!\\to\\\!Bλ​∥zA−zB∥2\\lambda\\lVert z\_\{A\}\{\-\}z\_\{B\}\\rVert^\{2\}A→BA\\\!\\to\\\!BB→AB\\\!\\to\\\!Ashared latentnon\-affine LNMaps supervised inℒ\\mathcal\{L\}:DA​\(zA\+ξA\)→xAD\_\{A\}\(z\_\{A\}\{\+\}\\xi\_\{A\}\)\{\\to\}x\_\{A\},DB​\(zB\+ξB\)→xBD\_\{B\}\(z\_\{B\}\{\+\}\\xi\_\{B\}\)\{\\to\}x\_\{B\}, and the alignment term\.The dashed cross\-family read\-outsA→BA\\\!\\to\\\!BandB→AB\\\!\\to\\\!Ado not appear in the loss\.Fig\. 1:Shared\-latent adapter\. Solid lines are the self\-reconstructions and the alignment that the loss includes, and dashed lines are the cross\-family read\-outs that the loss does not include\.Each encoder and decoder consists of an input linear projection, one residual block, and an output linear projection\. The residual block normalizes its input, applies a fully connected layerfc1\\mathrm\{fc\}\_\{1\}, the activation function GELU \(Gaussian Error Linear Unit\), and a fully connected layerfc2\\mathrm\{fc\}\_\{2\}in turn, and adds the result to its input, which is a pre\-norm configuration\. Both the latent width and the output width offc1\\mathrm\{fc\}\_\{1\}are40964096\. Because the weights offc2\\mathrm\{fc\}\_\{2\}are zero\-initialized, the residual block starts as the identity map, andEEandDDbegin training as linear maps\. A non\-affine LN lies between them, however, so a cross\-family read\-out as a whole is not linear even at initialization\. Zero initialization is a property at the start of training and does not mean that the trained map is linear\. Compared with the14\.2514\.25B parameters of the two parent models combined, the adapter has268\.5268\.5M parameters, a ratio of1\.9%1\.9\\%\.

The noiseξ\\xiis introduced to keep the latent from being dominated by a constant component that does not depend on the token\. We call this component the common mode\. The alignment term also becomes small whenzAz\_\{A\}andzBz\_\{B\}consist only of this constant component\. Adding noise just before decoding, however, makes it hard to recover the original residuals from a latent with little per\-token variation and thus raises the self\-reconstruction error\. The noise is a design that relies on this property to make the latent keep its per\-token variation\.

### III\-DConfigurations and Chimera Models

We call the layer at which the residual of the front model is converted by the adapter and passed to the back model the switch layer, the model that runs up to the switch layer the front model, and the model that runs after the switch layer the back model\. A combination of a front and a back model is called a configuration, including the case in which the same model is used for both; a model with a specified switch layer and a parent model alone are also called configurations\. The front\-back combinations compared in this study are of four kinds: RWKV self\-reconstruction control, RWKV→\\toPythia, Pythia self\-reconstruction control, and Pythia→\\toRWKV\. RWKV→\\toPythia and Pythia→\\toRWKV are cross\-family chimera models, and RWKV self\-reconstruction control and Pythia self\-reconstruction control are self\-reconstruction controls that pass through the same adapter at the same switch layer\. The switch layerLLis an index counted from00and denotes the last block that the front parent model runs\. Because block00is counted, the front model runsL\+1L\{\+\}1blocks\. In the notation of §[III\-B](https://arxiv.org/html/2609.38261#S3.SS2), the conversion at the switch layer is given by the following equation, where a tilde marks a residual converted into the coordinate system of the other model\.

h~BL​\[t\]\\displaystyle\\widetilde\{h\}\_\{B\}^\{L\}\[t\]=unstdBL​\(DB​\(LN⁡\(EA​\(xAL​\[t\]\)\)\)\),\\displaystyle=\\mathrm\{unstd\}\_\{B\}^\{L\}\\\!\\left\(D\_\{B\}\\\!\\left\(\\mathrm\{LN\}\(E\_\{A\}\(x\_\{A\}^\{L\}\[t\]\)\)\\right\)\\right\),\(2\)h~AL​\[t\]\\displaystyle\\widetilde\{h\}\_\{A\}^\{L\}\[t\]=unstdAL​\(DA​\(LN⁡\(EB​\(xBL​\[t\]\)\)\)\)\.\\displaystyle=\\mathrm\{unstd\}\_\{A\}^\{L\}\\\!\\left\(D\_\{A\}\\\!\\left\(\\mathrm\{LN\}\(E\_\{B\}\(x\_\{B\}^\{L\}\[t\]\)\)\\right\)\\right\)\.BlockL\+1L\{\+\}1ofBBreceivesh~BL​\[t\]\\widetilde\{h\}\_\{B\}^\{L\}\[t\], and blockL\+1L\{\+\}1ofAAreceivesh~AL​\[t\]\\widetilde\{h\}\_\{A\}^\{L\}\[t\]\. Equation \([2](https://arxiv.org/html/2609.38261#S3.E2)\) adds no training noise\. Execution has three steps: running blocks00toLLof the front parent model, converting the residual once, and running blocksL\+1L\{\+\}1to3131and the language\-model head of the back parent model, as shown in Fig\.[2](https://arxiv.org/html/2609.38261#S3.F2)\. The conversion is applied only once, at the switch layer, with no round trip at each block\. Every chimera model therefore runs3232blocks in total,L\+1L\{\+\}1in the front model and31−L31\{\-\}Lin the back model\. The error in the residual introduced by the conversion at the switch layer is called the conversion error\. Which model serves as the front model is called the connection direction\. In Pythia→\\toRWKV \(L=4L\{=\}4\), for example, the55blocks00to44of Pythia run first, followed by the2727blocks55to3131of RWKV\.

A configuration is denoted by the names of its front and back models, as in RWKV→\\toPythia, and the switch layer is added as in RWKV→\\toPythia \(L=4L\{=\}4\)\. Read\-outs, in contrast, are denoted by symbols such asA→BA\\\!\\to\\\!B\. The read\-out used at the switch layer of RWKV→\\toPythia isA→BA\\\!\\to\\\!B, that used by Pythia→\\toRWKV isB→AB\\\!\\to\\\!A, and those used by RWKV self\-reconstruction control and Pythia self\-reconstruction control areA→AA\\\!\\to\\\!AandB→BB\\\!\\to\\\!B, respectively\. In tables, RWKV is abbreviated as R and Pythia as P, and RWKV→\\toPythia and RWKV self\-reconstruction control are written, for example, as R→\\toP and R self\.

RWKV→\\toPythiaidsA0A\_\{0\}⋯\\cdotsALA\_\{L\}EA→zE\_\{A\}\\\!\\to z→DB\\to D\_\{B\}BL\+1B\_\{L\+1\}⋯\\cdotsB31B\_\{31\}BBheadlogitsPythia→\\toRWKVidsB0B\_\{0\}⋯\\cdotsBLB\_\{L\}EB→zE\_\{B\}\\\!\\to z→DA\\to D\_\{A\}AL\+1A\_\{L\+1\}⋯\\cdotsA31A\_\{31\}AAheadlogitsA=A=RWKV, no KV cacheB=B=Pythia, KV cacheFig\. 2:Configurations of the RWKV→\\toPythia and Pythia→\\toRWKV chimera models\.
### III\-EEvaluation Metrics

How well a residual is recovered is measured by the coefficient of determinationR2=1−SSE/SSTR^\{2\}=1\-\\mathrm\{SSE\}/\\mathrm\{SST\}\.SSE\\mathrm\{SSE\}is the sum of squared errors between the predicted and true residuals, andSST\\mathrm\{SST\}is the total sum of squares of the true residuals after subtracting the per\-dimension token mean\. A value of11means perfect recovery,00means the same performance as always predicting the per\-dimension mean, and a negative value means worse than this mean predictor\. We measure four read\-outs: the cross\-familyA→BA\\\!\\to\\\!BandB→AB\\\!\\to\\\!Aand the self\-reconstructionsA→AA\\\!\\to\\\!AandB→BB\\\!\\to\\\!B\. The latter two are controls that show the information lost by encoding and decoding\. Below, theR2R^\{2\}of a read\-out is called its read\-out accuracy\.

R2R^\{2\}is computed per layer over the evaluation tokens and all40964096dimensions together\. BecauseSST\\mathrm\{SST\}is the total variation after removing the per\-dimension token mean, dimensions with larger variance contribute more\. The alignment of the latent spaces is measured by the centered correlationρctr\\rho\_\{\\mathrm\{ctr\}\}, the Pearson correlation coefficient ofzA,zBz\_\{A\},z\_\{B\}flattened into one dimension after subtracting the token mean from each dimension\. Values close to11mean that the two latents vary in the same direction, and00means no linear correlation\. We further define the varying fraction, the fraction of the per\-token varying component of the latent, asf=∥z−z¯∥2/∥z∥2f=\\lVert z\-\\overline\{z\}\\rVert^\{2\}/\\lVert z\\rVert^\{2\}, wherez¯\\overline\{z\}is the per\-dimension token mean within a layer andffis computed separately forAAandBBand averaged\. Each metric is computed per layer, and we report the simple mean over the3232layers\.

There are two sets of evaluation data, both taken from positions that do not overlap the training sequences\. The main values in the text are computed on sequences of112112tokens,1,0001\{,\}000sequences or112,000112\{,\}000tokens in total\. These data are also used to adjust the learning rate and to select the checkpoint \(§[III\-F](https://arxiv.org/html/2609.38261#S3.SS6)\)\. The per\-layer analysis and the shuffle control, which re\-pairs the tokens at random, use400400sequences of the same length,44,80044\{,\}800tokens in total\. Because the evaluation data differ, the3232\-layer means of the per\-layer analysis differ slightly from the main values; both are given in §[IV\-A](https://arxiv.org/html/2609.38261#S4.SS1)\. The affine control for accuracy evaluation in §[IV\-C](https://arxiv.org/html/2609.38261#S4.SS3)is fitted on40,09640\{,\}096rows of standardized residuals taken from the part of Alpaca not used to train the adapter\. Each row is the residual vector at one token position\. These40,09640\{,\}096rows and the sequences used for the linearity evaluation and the intervention in §[IV\-C](https://arxiv.org/html/2609.38261#S4.SS3)are leading parts of the main evaluation data, taken from the same positions by the same procedure, and are contained in it\. The fittedR2R^\{2\}of the all\-layer correspondence expresses how well the representation of one layer can be predicted linearly from that of another\. A linear map is fitted by ridge regression, that is, least squares with L2 regularization, to the22,40022\{,\}400rows shared by each pair of layers, which come from200200sequences of112112tokens taken from the same positions as the main evaluation data\. The same samples are used for fitting and evaluation, so for a40964096\-dimensional linear map the value is easily inflated\.

The language ability of the chimera models is evaluated in three ways\. First, English prompts are given without modification, and generation examples are obtained by deterministic argmax decoding at temperature00, which selects the most probable token at each step\. Generation is limited to4040new tokens and stops early only when the model outputs EOS \(end\-of\-sequence\), the special token that marks the end of a sequence\. Second, the log\-likelihood of each option of a multiple\-choice question is computed, the most likely option is taken as the answer, and accuracy is measured\. The question sets are ARC\-Easy\[[30](https://arxiv.org/html/2609.38261#bib.bib25)\], the subset of the ARC \(AI2 Reasoning Challenge\) dataset of science exam questions for elementary and middle school students that excludes the hard questions both a retrieval\-based and a word co\-occurrence baseline answered incorrectly, and SciQ\[[31](https://arxiv.org/html/2609.38261#bib.bib26)\]\. Below, ARC\-Easy is abbreviated as ARC\. Both mostly have 4 options, and random choice gives an accuracy of about25%25\\%\.

We use all23762376test questions of ARC and all10001000test questions of SciQ\. SciQ is evaluated in closed\-book form, without its support passages, so our SciQ values cannot be compared directly with published values obtained under the standard protocol that provides those passages\. Two scores are computed from the log\-likelihoods of the option tokens conditioned on the question, that is, of the tokens after the prompt: their sum and their average\. Options differ in length and the sum tends to be smaller for longer options, so the text mainly uses the latter, the length\-normalized score\. Differences between configurations are computed from paired data on the same questions, so they are evaluated with McNemar’s exact test for paired binary data and with a paired bootstrap that estimates the variability of the difference by resampling pairs of the same questions\. The text and tables give McNemarpp\-values, and bootstrap confidence intervals of the differences are included in the released data\. Accuracy differences are expressed in percentage points, called points below\.

Third, the next\-token cross\-entropy is measured on Alpaca and on WikiText\-103\[[32](https://arxiv.org/html/2609.38261#bib.bib27)\], a corpus built from good\-quality Wikipedia articles and called WikiText below, to compare language\-modeling performance inside and outside the domain of text used to train the adapter, called the training domain below\.

### III\-FTraining Setup

The adapter is trained on the residuals of2,0752\{,\}075sequences of112112tokens taken from the beginning of the Alpaca training split\. The distinct token positions number about232232k, and the residual pairs number7\.447\.44M over the3232layers\. This set of pairs was iterated about228228times\. For optimization we used AdamW, which decouples weight decay from Adam \(Adaptive Moment Estimation\), with an effective batch size of40964096, an initial learning rate of10−310^\{\-3\}, and a minimum learning rate of10−510^\{\-5\}\. At each iteration one of the3232layers is chosen uniformly at random, and an effective batch is drawn from the residual pairs of the chosen layer\. Gradients are clipped so that their norm does not exceed1\.01\.0\. Every1,0001\{,\}000iterations,A→BA\\\!\\to\\\!Bwas measured on the evaluation data \(§[III\-E](https://arxiv.org/html/2609.38261#S3.SS5)\), and a reference value was updated only by an evaluation that exceeded it by more than a relative10−310^\{\-3\}; the learning rate was multiplied by0\.50\.5each time44consecutive evaluations brought no update\. We ran416,000416\{,\}000iterations with random seed00and used the checkpoint at iteration415,000415\{,\}000, whereA→BA\\\!\\to\\\!Bon the evaluation data was best, for all subsequent evaluations\. The checkpoint was not reselected based onB→AB\\\!\\to\\\!Aor on multiple\-choice results\. Details of the implementation and the released artifacts are given in Appendix[A](https://arxiv.org/html/2609.38261#A1)\.

## IVResults

We first examine whether the representations of corresponding layers align, and measure how well each model’s residuals can be recovered from the resulting representation\. We then analyze the nonlinearity of the adapter, and finally evaluate the accuracy, inference memory, performance outside the training domain, and generation examples of the composed chimera models\.

### IV\-AShared Representation Between Corresponding Layers

The adapter trained on the residuals of both models shows a centered correlation ofρctr=0\.901\\rho\_\{\\mathrm\{ctr\}\}\\\!=\\\!0\.901on text not used for training\. In addition to this value, we take as evidence that a shared representation exists the fact that the cross\-family read\-out accuracy is0\.5590\.559forA→BA\\\!\\to\\\!Band0\.5900\.590forB→AB\\\!\\to\\\!Aand that both fall sharply under the shuffle control described next\. The self\-reconstruction read\-out accuracies,A→A​0\.695A\\\!\\to\\\!A\\ 0\.695andB→B​0\.711B\\\!\\to\\\!B\\ 0\.711, indicate how much information is lost by encoding and decoding\.

This shared representation is based on per\-token correspondence rather than on overlapping distributions\. When the row order ofAAis kept and the residual rows ofBBare re\-paired under a single random permutation, on the evaluation data for the per\-layer analysisρctr\\rho\_\{\\mathrm\{ctr\}\}falls as→0\.0000\.901\\\!\\to\\\!0\.000, and the cross\-family read\-out accuracy falls as→−0\.5920\.561\\\!\\to\\\!\-0\.592forA→BA\\\!\\to\\\!Band as→−0\.6090\.592\\\!\\to\\\!\-0\.609forB→AB\\\!\\to\\\!A\. The self\-reconstruction read\-out accuracies do not change, by construction\. Neither cross\-family read\-out therefore succeeds without the correct token correspondence\.

Same\-index layer pairs, however, are not intrinsically special\. As Fig\.[3](https://arxiv.org/html/2609.38261#S4.F3)shows, comparing the unstandardized residualshhdirectly, without the adapter, over all32×3232\{\\times\}32layer pairs gives a small linear CKA overall\. The mean of the diagonal elements, which are the pairs of layers with the same index, is0\.050\.05, and the mean over the other pairs is0\.030\.03\. LayerjjofAAis most similar to layerjjofBBin only22of3232rows\. For1414of the3232layers ofAAthe most similar layer ofBBis layer00, and for1212it is the last layer\. Same\-index layers are not special in linear predictability either\. Fitting a linear map to every layer pair gives fittedR2R^\{2\}values of0\.620\.62–0\.940\.94, high for almost every pair\. These values are measured on the rows used for fitting, however, and cannot be compared with values evaluated on rows not used for fitting, such as those of the direct affine maps in §[IV\-C](https://arxiv.org/html/2609.38261#S4.SS3)\. The mean fittedR2R^\{2\}is0\.870\.87on the diagonal and0\.840\.84off the diagonal, and the same\-index layer is best in only33of3232rows\. The valueρctr=0\.901\\rho\_\{\\mathrm\{ctr\}\}\\\!=\\\!0\.901is therefore an alignment obtained by training on the chosen pairs of same\-index layers, not a correspondence inherent in layers that share an index\. In other words, this study is a case study that connects same\-index layer pairs in a setting where any layer pair can be predicted by linear regression to some degree\.

![Refer to caption](https://arxiv.org/html/2609.38261v1/fig_layer_cka_en.png)Fig\. 3:Correspondence over all32×3232\{\\times\}32layer pairs of the unstandardized residualshhtaken without the adapter\. Left: linear CKA\. Right: fittedR2R^\{2\}of per\-layer linear maps\. Dashed lines mark same\-index layers, and the two panels use different color scales\.The Pearson correlation coefficient ofzA,zBz\_\{A\},z\_\{B\}before centering, called the raw correlation below, isρraw=0\.965\\rho\_\{\\mathrm\{raw\}\}=0\.965, which is higher than the centered correlation0\.9010\.901\. The difference is explained by the common mode\. With the varying fractionffdefined in §[III\-E](https://arxiv.org/html/2609.38261#S3.SS5), the following holds under a layer normalization with no per\-element coefficients when the varying fractions ofAAandBBare equal and their common modes are parallel\.

ρraw=\(1−f\)\+f​ρctr\.\\rho\_\{\\mathrm\{raw\}\}=\(1\-f\)\+f\\,\\rho\_\{\\mathrm\{ctr\}\}\.\(3\)Hereffis the mean of the varying fractions ofAAandBB\. At evaluations after training has progressed, Equation \([3](https://arxiv.org/html/2609.38261#S4.E3)\) holds to within about×10−43\\\!\\times\\\!10^\{\-4\}\. At iteration00, however, the error is×10−26\.6\\\!\\times\\\!10^\{\-2\}\. The checkpoint at iteration415,000415\{,\}000hasf=0\.350f=0\.350, so the token\-independent common mode accounts for65%65\\%of the squared norm of the latent\. Equation \([3](https://arxiv.org/html/2609.38261#S4.E3)\) reproduces the raw correlation exactly as\(1−0\.350\)\+0\.350×0\.901=0\.965\(1\{\-\}0\.350\)\+0\.350\\times 0\.901=0\.965, which shows that most of the raw correlation is due to the common mode, which contains no information\. We therefore use the centered correlation as the measure of alignment\.

Note thatffis a second moment, not an amount of information\.f→0f\\\!\\to\\\!0means that the latent has degenerated to a constant and contains no per\-token information, whereas a largeffonly shows that the latent varies in direction\. Whether the varying component contains information is judged by the read\-outs\. The read\-out measure is a centeredR2R^\{2\}, which no token\-independent predictor can raise above00\. HenceB→B=0\.711B\\\!\\to\\\!B=0\.711cannot be obtained from a constant prediction, which is consistent with the common mode accounting for65%65\\%of the squared norm of the latent\.

### IV\-BSelf\-Reconstruction Compared with Cross\-Family Read\-Outs

Fig\.[4](https://arxiv.org/html/2609.38261#S4.F4)shows the relation among the five measures described in §[IV\-A](https://arxiv.org/html/2609.38261#S4.SS1)\.ρctr\\rho\_\{\\mathrm\{ctr\}\}is a correlation coefficient and the other four are coefficients of determination, so they cannot be compared on the same scale\. Among the cross\-family read\-out accuracies,B→AB\\\!\\to\\\!A, used by Pythia→\\toRWKV, slightly exceedsA→BA\\\!\\to\\\!B, used by RWKV→\\toPythia, so the RWKV residuals are easier to recover from the latent obtained with the Pythia encoder\. Both read\-outs fall below the self\-reconstruction that uses the same decoder and differs only in the input latent\. For the read\-out accuracies,A→B≤B→BA\\\!\\to\\\!B\\leq B\\\!\\to\\\!BandB→A≤A→AB\\\!\\to\\\!A\\leq A\\\!\\to\\\!Ahold in all3232layers\.

latent alignmentρctr=0\.901\\rho\_\{\\mathrm\{ctr\}\}=0\.901BBself\-reconstructionR2​\(B→B\)=0\.711R^\{2\}\(B\\\!\\to\\\!B\)=0\.711cross\-family read\-outR2​\(A→B\)=0\.559R^\{2\}\(A\\\!\\to\\\!B\)=0\.559AAself\-reconstructionR2​\(A→A\)=0\.695R^\{2\}\(A\\\!\\to\\\!A\)=0\.695cross\-family read\-outR2​\(B→A\)=0\.590R^\{2\}\(B\\\!\\to\\\!A\)=0\.590reconstructreconstructswap in theother latentswap in theother latentFig\. 4:Relation between the shared latent space and the four read\-outs\. Read\-outs of the same color share a decoder\.This ordering is observed in experiments rather than guaranteed by theory; still, in both directions substantial error already exists at the self\-reconstruction stage, and improving the cross\-family part does not remove the self\-reconstruction error\.

Atdz=d=4096d\_\{z\}\{=\}d\{=\}4096the latent is as wide as the residual, and the non\-affine LN removes only two degrees of freedom per token, the mean and the norm\. Latent width or rank is therefore unlikely to constrain reconstruction strongly\. Even so,B→BB\\\!\\to\\\!Bis only0\.7110\.711\. Whether this is an upper bound set by the noiseσ\\sigmais unknown\. We have also not isolated whether the error arises in the encoder, the latent normalization, the decoder, or the training noise, and what limits the reconstruction accuracy remains unresolved\.

### IV\-CNonlinearity of the Adapter

As described in §[III\-C](https://arxiv.org/html/2609.38261#S3.SS3),EEandDDof the adapter begin training as linear maps\. This is a property before training, however, not after415,000415\{,\}000iterations\. We therefore examine the linearity of the adapter at this checkpoint with three affine maps that play different roles\. Maps \(i\) and \(ii\) were measured for both RWKV→\\toPythia and Pythia→\\toRWKV on the same55representative layers\{4,10,16,22,28\}\\\{4,10,16,22,28\\\}and the same fitting and evaluation rows\. The55representative layers were taken every66layers from layer44to layer2828\. The rows are standardized residuals of900900sequences of112112tokens taken from positions of Alpaca not used to train the adapter; for each layer the first50,40050\{,\}400rows are used for fitting and the last50,40050\{,\}400rows for evaluation\.

\(i\) The imitation affine map is the affine map that best approximates the output of the trained adapter, and it is used to measure how linear the adapter is\. The fraction of the adapter’s output variance that it explains is86\.6%86\.6\\%for RWKV→\\toPythia and88\.0%88\.0\\%for Pythia→\\toRWKV\. The remaining13\.4%13\.4\\%and12\.0%12\.0\\%therefore cannot be explained on the evaluation rows by the affine map fitted on the first half of the rows, and the trained map is nonlinear in both directions\.

\(ii\) The direct affine map predicts, without the adapter, the standardized output residual from the standardized input residual for each layer and direction; it is the best affine map fitted by ridge regression\. The regularization coefficient is10−310^\{\-3\}timestr⁡\(X⊤​X\)/\(d\+1\)\\mathrm\{tr\}\(X^\{\\top\}X\)/\(d\{\+\}1\)for the input matrixXXwith an intercept column appended\. This map measures how much cross\-family prediction a low\-capacity map achieves\. On the evaluation rows, the best direct affine map gives0\.4870\.487for RWKV→\\toPythia and0\.5350\.535for Pythia→\\toRWKV\. On the same rows the adapter gives0\.5580\.558for RWKV→\\toPythia and0\.5760\.576for Pythia→\\toRWKV, so the nonlinear map adds\+0\.071\+0\.071and\+0\.041\+0\.041\. These values were evaluated on the55representative layers, and both the layers and the rows differ from those of the3232\-layer means0\.5590\.559and0\.5900\.590in §[IV\-A](https://arxiv.org/html/2609.38261#S4.SS1)\. That even a low\-capacity affine map predicts about half of the variance of the other model’s residual in both directions,87%87\\%and93%93\\%of the adapter’s values, is consistent with the conclusion that a shared representation exists\.

\(iii\) The affine control for accuracy evaluation embeds an affine map like \(ii\) in the chimera model and serves as a control for comparing multiple\-choice accuracy with the adapter\. It was fitted for each switch layer on the rows described in §[III\-E](https://arxiv.org/html/2609.38261#S3.SS5), by ridge regression with an intercept and the same regularization coefficient as \(ii\)\. As Table[I](https://arxiv.org/html/2609.38261#S4.T1)shows, replacing the adapter of RWKV→\\toPythia with this control lowers accuracy by88–2323points\. For Pythia→\\toRWKV, by contrast, the difference is small and lies within−3\.4\-3\.4–\+2\.0\+2\.0points over the88conditions\. AtL=4L\{=\}4on ARC the adapter is2\.02\.0points higher \(p=0\.005p\{=\}0\.005\), whereas atL=8L\{=\}8on SciQ the affine control is3\.43\.4points higher \(p=0\.003p\{=\}0\.003\)\. The two maps differ in training objective, data, and regularization, so we do not treat this difference between directions as a causal effect of nonlinearity\.

TABLE I:Length\-normalized accuracy \(%\) of the adapter \(Ad\) and of the affine control for accuracy evaluation \(Af\)\. R→\\toP: RWKV→\\toPythia; P→\\toR: Pythia→\\toRWKV\.Next, we intervene to disable the nonlinearity of the adapter\. To suppress the confounds above, in each residual block of the same trained adapter we scale by a coefficientα\\alphaonly the output of the branch that normalizes the input and appliesfc1\\mathrm\{fc\}\_\{1\}, GELU, andfc2\\mathrm\{fc\}\_\{2\}\. This branch is the only learned nonlinearity in the adapter, and we call its output the nonlinear part\.α=1\\alpha\{=\}1is the original map, andα=0\\alpha\{=\}0is the same map with the nonlinear part removed\. Sweepingα=0,\.25,\.5,\.75,1\\alpha=0,\.25,\.5,\.75,1for the mean cross\-family read\-out accuracy of the55representative layers on the evaluation data for the per\-layer analysis \(§[III\-E](https://arxiv.org/html/2609.38261#S3.SS5)\) gives−0\.41,−0\.75,0\.35,0\.52,0\.56\-0\.41,\-0\.75,0\.35,0\.52,0\.56for RWKV→\\toPythia and−0\.30,−0\.66,0\.36,0\.53,0\.58\-0\.30,\-0\.66,0\.36,0\.53,0\.58for Pythia→\\toRWKV\. Removing the nonlinear part lowers the read\-out accuracy to negative values in both directions, and the dependence onα\\alphais non\-monotonic\. Even atα=0\\alpha\{=\}0the non\-affine LN remains, so the map does not become entirely linear\. The input and output linear projections of the encoder and the decoder were trained jointly with the nonlinear part, and in this trained adapter the nonlinear part is indispensable to both the RWKV→\\toPythia and Pythia→\\toRWKV read\-outs\.

TABLE II:Change in length\-normalized accuracy \(%\) when the nonlinear part is disabled,α:→0\\alpha:1\\\!\\to\\\!0\. R→\\toP: RWKV→\\toPythia; P→\\toR: Pythia→\\toRWKV\.The effect of the nonlinear part in Table[II](https://arxiv.org/html/2609.38261#S4.T2)is\+11\.1\+11\.1–\+39\.6\+39\.6points for RWKV→\\toPythia and\+11\.2\+11\.2–\+36\.6\+36\.6points for Pythia→\\toRWKV, and McNemar’s exact test on the same questions givesp<2×10−9p<2\{\\times\}10^\{\-9\}in all1616comparisons\. Removing the nonlinear part from this trained adapter therefore lowers the task accuracy as well as the read\-out accuracy for both RWKV→\\toPythia and Pythia→\\toRWKV\. This is, however, a property of removing the nonlinear part from a jointly trained map, and it does not show that nonlinearity is generally necessary for connecting families\. We did not compare against a linear adapter trained for the same objective, and the accuracy difference between the affine control and the adapter is small for Pythia→\\toRWKV \(Table[I](https://arxiv.org/html/2609.38261#S4.T1)\)\.

### IV\-DAccuracy Compared with the Parent Models

This subsection compares the chimera models with the two frozen parent models to examine how much accuracy is retained relative to the stronger parent and whether the weaker parent is exceeded\. The evaluation method is as described in §[III\-E](https://arxiv.org/html/2609.38261#S3.SS5)\.

TABLE III:Length\-normalized accuracy \(%\) of the four configurations\. The maximum of each row and task is in bold\. R self: RWKV self\-reconstruction control; P self: Pythia self\-reconstruction control; R→\\toP: RWKV→\\toPythia; P→\\toR: Pythia→\\toRWKV\.TABLE IV:Accuracy differences \(points\) between the main configurations and McNemar exact\-testpp\-values in parentheses\. R→\\toP: RWKV→\\toPythia; P→\\toR: Pythia→\\toRWKV\.Table[III](https://arxiv.org/html/2609.38261#S4.T3)gives the accuracies of the four configurations, and Table[IV](https://arxiv.org/html/2609.38261#S4.T4)gives paired tests for the main comparisons\. In Table[IV](https://arxiv.org/html/2609.38261#S4.T4), “vs RWKV” is the accuracy difference between each configuration and RWKV alone, and “Dir\.” is the difference between Pythia→\\toRWKV and RWKV→\\toPythia at the same switch layer; both are based on the length\-normalized score\. The comparisons with Pythia alone are omitted from the table\. The main results are the following five points\.

First, even though the front and back models come from different families, both RWKV→\\toPythia and Pythia→\\toRWKV answer the questions, and every configuration exceeds the accuracy of random choice, about25%25\\%, by a wide margin\. Second, the chimera models are significantly less accurate than the stronger parent Pythia, whose accuracy is ARC65\.3%65\.3\\%and SciQ81\.0%81\.0\\%\(Table[V](https://arxiv.org/html/2609.38261#S4.T5)\), in all1616conditions over both directions, all switch layers, and both tasks \(p≤0\.003p\\leq 0\.003\)\. Third, among the chimera models, the only one that significantly exceeds RWKV, whose accuracy is ARC61\.8%61\.8\\%and SciQ67\.0%67\.0\\%, is RWKV→\\toPythia \(L=4L\{=\}4\) on SciQ, and Pythia→\\toRWKV does not significantly exceed RWKV at any switch layer\. The advantage of RWKV→\\toPythia \(L=4L\{=\}4\) appears only with the length\-normalized score; with the summed likelihood score, the differences from RWKV alone are−2\.7\-2\.7points on SciQ for RWKV→\\toPythia \(L=4L\{=\}4\), and−4\.6\-4\.6points on ARC and−5\.7\-5\.7points on SciQ for Pythia→\\toRWKV \(L=4L\{=\}4\)\. Fourth, the effect of the switch depth depends on the direction\. RWKV→\\toPythia loses accuracy monotonically as the switch layer deepens, whereas Pythia→\\toRWKV is non\-monotonic and substantially exceeds RWKV→\\toPythia at the same switch layer forL=16,24L\{=\}16,24\. On the evaluation data for the per\-layer analysis, the cross\-family read\-out accuracies at the same four layers are0\.570/0\.555/0\.575/0\.5670\.570/0\.555/0\.575/0\.567for RWKV→\\toPythia and0\.552/0\.517/0\.604/0\.6080\.552/0\.517/0\.604/0\.608for Pythia→\\toRWKV, so the accuracy ranking cannot be predicted from read\-out accuracy alone\. Fifth, the loss from encoding and decoding also appears in the accuracy of the self\-reconstruction controls: the difference from Pythia alone is significant for Pythia self\-reconstruction control at every switch layer on both tasks, and the difference from RWKV alone is significant for RWKV self\-reconstruction control on both tasks atL=16,24L\{=\}16,24\. AtL=8L\{=\}8it is significant only on ARC \(p=0\.006p\{=\}0\.006\)\. The differences from the self\-reconstruction controls, however, are not due to conversion error alone\. With Pythia as the back model, accuracy decreases in the order of Pythia alone, Pythia self\-reconstruction control, and RWKV→\\toPythia, and the difference between Pythia self\-reconstruction control and RWKV→\\toPythia is33–1212points on ARC and44–1919points on SciQ\. With RWKV as the back model, in contrast, Pythia→\\toRWKV exceeds RWKV self\-reconstruction control at deep switch layers, by\+2\.7/\+7\.4\+2\.7/\+7\.4points on ARC and\+8\.2/\+12\.0\+8\.2/\+12\.0points on SciQ atL=16/24L\{=\}16/24\. A difference from a cross\-family configuration also contains the difference in ability between the front parent models, so it cannot be read as the loss caused by conversion\.

### IV\-EKV Cache Reduction

For each layer and token, Pythia’s KV cache holds two40964096\-dimensional vectors, the key and the value, in FP16, totaling1616KiB, or512512KiB over3232layers\. RWKV has no KV cache but instead keeps a recurrent state of6464KiB per layer that does not depend on the context length\. This value is computed from the reference implementation, which keeps22of its55state vectors in FP16 and33in 32\-bit floating point \(FP32\)\. The KV cache sizes in Table[V](https://arxiv.org/html/2609.38261#S4.T5)exclude the recurrent state\. RWKV→\\toPythia keeps31−L31\{\-\}LTransformer layers, and Pythia→\\toRWKV keepsL\+1L\{\+\}1\. The reduction applies to the KV cache, not to the resident weights\. The adapter itself occupies0\.500\.50GiB in FP16 and about0\.250\.25GiB even when only one direction is kept in memory, which is comparable to the reduced KV cache\. Even when the unused blocks are physically removed, the resident weights of a chimera model, including the adapter, are13\.4913\.49–14\.5514\.55GiB, about the same as the13\.7713\.77GiB of RWKV alone and the12\.7712\.77GiB of Pythia alone\. The output of the implementation with the blocks removed matched that of the implementation without removal\.

TABLE V:Accuracy and inference memory\. KV@4k: KV cache size at context length40964096\. P alone: Pythia; R alone: RWKV; R→\\toP: RWKV→\\toPythia; P→\\toR: Pythia→\\toRWKV\.Comparing RWKV→\\toPythia and Pythia→\\toRWKV, Pythia→\\toRWKV \(L=4L\{=\}4\) reduces the Transformer layers from27→527\\to 5relative to RWKV→\\toPythia \(L=4L\{=\}4\) while keeping the two\-task mean accuracy at an equivalent level,65\.25%→65\.70%65\.25\\%\\to 65\.70\\%\. Because the ranking depends on the task, Fig\.[5](https://arxiv.org/html/2609.38261#S4.F5)shows the trade\-off between KV cache reduction and accuracy on SciQ\. We call the non\-dominated set the set of configurations for which no other configuration is at least as good in both measures and better in at least one of them\.

SciQaccuracy/ %KV cache reduction / %020406080100556575Pythia aloneL=4L=4L=8L=8L=16L=16L=24L=24L=4L=4L=8L=8L=16L=16L=24L=24RWKV alone∙\\bulletRWKV→\\toPythia, RWKV front■\\blacksquarePythia→\\toRWKV, Pythia frontSciQ non\-dominated setFig\. 5:SciQ accuracy and KV cache reduction of RWKV→\\toPythia and Pythia→\\toRWKV\. The green line connects the configurations in the non\-dominated set\.Adding the configurations with Pythia as the front model yields Pythia→\\toRWKV \(L=4L\{=\}4\), which has a high reduction rate and cannot be obtained from the configurations with RWKV as the front model alone\. RWKV alone, on the other hand, needs no KV cache, and its accuracy does not differ significantly from that of Pythia→\\toRWKV \(L=4L\{=\}4\) \(§[IV\-D](https://arxiv.org/html/2609.38261#S4.SS4)\)\. The advantage of the method is therefore that it creates new trade\-off points between frozen parent models, not that new configurations uniformly exceed the parent models\.

If the only goal is to reduce the KV cache, the same reduction is also obtained by running only the first blocks of Pythia and connecting their output to the final layer normalization and the LM head\. We call this method early exit\. We therefore compared the accuracy of each chimera model with that of an early exit that keeps the same number of Transformer blocks \(Table[VI](https://arxiv.org/html/2609.38261#S4.T6)\)\. In the55configurations that reduce the Transformer layers to1717or fewer, the chimera model significantly exceeds the early exit on both tasks\. Pythia→\\toRWKV \(L=4L\{=\}4\) scores62\.2%62\.2\\%against26\.8%26\.8\\%on ARC and69\.2%69\.2\\%against30\.5%30\.5\\%on SciQ, compared with the early exit that stops after the first55blocks of Pythia\. In the33configurations that keep2323or more layers, by contrast, the differences are small; only ARC for RWKV→\\toPythia \(L=8L\{=\}8\), which keeps2323layers, is significant, and RWKV→\\toPythia \(L=4L\{=\}4\), which keeps2727layers, does not differ from the early exit\. However, because the early exit runs fewer blocks and keeps fewer resident weights at the same KV cache size, this comparison does not equalize computation\.

TABLE VI:Differences in length\-normalized accuracy \(points, chimera model minus early exit\) against a Pythia early exit that keeps the same number of Transformer layers, with McNemar exact\-testpp\-values\. Transformer: number of Transformer layers kept\. R→\\toP: RWKV→\\toPythia; P→\\toR: Pythia→\\toRWKV\.As for inference time, with the implementation in the Hugging Face Transformers library used in this study and the two parent models in FP16 placed on22NVIDIA V100 GPUs, we compared the time to process the whole input sequence at once, taking the median of33measurements\. At context length512512, RWKV alone took6,5386\{,\}538ms and Pythia alone9999ms, a factor of about6666\. This gap arises because the implementation used runs the RWKV recurrence sequentially, token by token\. At context length20482048the times were25,93325\{,\}933ms and385385ms, a factor of about6767\. For chimera models with the unused layers removed, configurations of the same direction take longer the more RWKV blocks they run; at context length512512, RWKV→\\toPythia \(L=4L\{=\}4\), which runs55RWKV blocks, took1,1281\{,\}128ms, and Pythia→\\toRWKV \(L=4L\{=\}4\), which runs2727RWKV blocks, took5,6525\{,\}652ms\. These values depend on the implementation used for the measurement and do not compare the computational cost of the architectures\.

### IV\-FEffect of Domain Shift

This subsection examines how performance changes when the evaluated text comes from a domain different from the training domain, a mismatch called domain shift below\. The accuracies above were measured on short questions close to instruction format\. Because a smaller KV cache matters for long contexts, we vary the context length and measure language\-modeling performance on Alpaca, inside the training domain, and on WikiText, outside it\. As the measure we use perplexity, the exponential of the next\-token cross\-entropy, which is better when smaller\. Fig\.[6](https://arxiv.org/html/2609.38261#S4.F6)shows the results at context length512512\. The perplexities of the parent models are3\.423\.42on Alpaca and11\.3311\.33on WikiText for RWKV, and5\.195\.19and16\.2216\.22for Pythia\. Alpaca, however, overlaps the instruction data of both parents, so the parents’ Alpaca perplexities may partly reflect exposure during instruction tuning\. RWKV→\\toPythia \(L=4L\{=\}4\) gives6\.406\.40on Alpaca and93\.8793\.87on WikiText, and Pythia→\\toRWKV \(L=4L\{=\}4\) gives3\.853\.85and45\.0045\.00; Pythia→\\toRWKV \(L=4L\{=\}4\) is lower than RWKV→\\toPythia \(L=4L\{=\}4\) in both domains, but on WikiText it is well above those of the parent models\. Extending the context length to20482048gives3\.503\.50on Alpaca and34\.7834\.78on WikiText for Pythia→\\toRWKV \(L=4L\{=\}4\), so performance does not degrade and even improves\. The main cause of the degradation is therefore domain shift, not context length\. Pythia→\\toRWKV is less affected by domain shift than RWKV→\\toPythia, but the WikiText perplexity of34\.7834\.78at context length20482048is4\.24\.2times the8\.278\.27of RWKV, the parent model with the lower perplexity at the same context length, and remains high\.

We also compared the adapter with a control that places at the switch layer an affine map fitted by ridge regression with an intercept, as in \(iii\) of §[IV\-C](https://arxiv.org/html/2609.38261#S4.SS3)\. This control is refitted for each switch layer and each evaluated domain on its first39,93639\{,\}936tokens, which are not used for evaluation; it is thus fitted even to WikiText, on which the adapter was not trained, so this comparison is unfavorable to the adapter\. For Pythia→\\toRWKV, the WikiText perplexity of the affine control is lower than that of the adapter at every switch layer and context length; for Pythia→\\toRWKV \(L=4L\{=\}4\) it is21\.7321\.73against45\.0045\.00at context length512512and15\.7415\.74against34\.7834\.78at context length20482048\. On Alpaca the two are nearly equal,3\.863\.86against3\.853\.85for Pythia→\\toRWKV \(L=4L\{=\}4\) at context length512512\. For RWKV→\\toPythia, in contrast, the WikiText perplexity of the affine control is at least that of the adapter at every switch layer and context length\. With this caveat, the result that the affine control is better for Pythia→\\toRWKV suggests that part of the degradation outside the training domain may arise from the conversion by the adapter, which was trained only on Alpaca\.

perplexity / log scale31030100300RWKVPythiaRWKV→\\toPythiaL=4L=4Pythia→\\toRWKVL=4L=4in\-domain Alpacaout\-of\-domain WikiTextFig\. 6:Perplexity on Alpaca and WikiText at context length512512\. The vertical axis is logarithmic\.
### IV\-GGeneration Examples of the Chimera Models

The adapter of the checkpoint at iteration415,000415\{,\}000was inserted atL=4L\{=\}4in both RWKV→\\toPythia and Pythia→\\toRWKV, and responses to the same four English prompts were generated by argmax decoding under the conditions of §[III\-E](https://arxiv.org/html/2609.38261#S3.SS5)\. Two of the prompts use instruction format, which separates the instruction and the response with the headings “\#\#\# Instruction:” and “\#\#\# Response:”, and two use completion format, which asks the model to continue a sentence\.L=4L\{=\}4is the switch layer at which the two\-task mean accuracy is highest among the four switch layers in each direction, and it is the common switch layer at which both directions belong to the non\-dominated set in the plane of KV cache reduction and accuracy on SciQ \(Fig\.[5](https://arxiv.org/html/2609.38261#S4.F5)\)\. This choice, however, was made after seeing the results of the evaluation tasks\.

TABLE VII:Generation examples of the two parent models and of RWKV→\\toPythia and Pythia→\\toRWKV atL=4L\{=\}4\.Prompt \(verbatim\)Pythia aloneRWKV→\\toPythia \(L=4L\{=\}4\)RWKV alonePythia→\\toRWKV \(L=4L\{=\}4\)\#\#\# Instruction:What is the capital of France?\#\#\# Response:“The capital of France is Paris\.”“The capital of France is Paris\.” ✓“Paris\.”“Paris\.” ✓The three primary colors are“red, yellow, and blue\. These colors are often used in combination to create a wide variety of colors\.”“red, yellow, and blue\.” ✓“red, blue, and yellow\.” then drifts into a Q&A asking for green, the opposite color of red“red, blue, and yellow\.” then drifts into a Bob/Alice dialogue about greenQuestion: Water is made of hydrogen andAnswer:“Hydrogen is made of hydrogen and oxygen\.”“Hydrogen\.” wrong“hydrogen\.” wrong“hydrogen\.” wrong\#\#\# Instruction:Explain what a black hole is in one sentence\.\#\#\# Response:“A black hole is a region of spacetime where gravity is so strong that nothing, not even light, can escape its pull\.”“A black hole is a region in space where gravity is so strong that nothing, not even light, can escape its pull\.” ✓“A black hole is a region of space where the gravitational pull is so strong that nothing, not even light, can escape\.”“A black hole is a type of celestial object that is formed when a star collapses and collapses into a black hole\.” circularThese examples show that both RWKV→\\toPythia and Pythia→\\toRWKV can generate syntactically well\-formed output\. Both directions answer the capital of France correctly and begin listing the primary colors correctly\. RWKV→\\toPythia also answers the black\-hole instruction correctly in one sentence, so generation that follows the instruction is achieved\. On the other hand, content or instruction format can be lost even when fluency is kept\. Both directions answer “hydrogen” wrongly in the completion about the composition of water, and Pythia→\\toRWKV drifts into a dialogue after answering the primary colors correctly and explains the black hole circularly\. Current chimera models can therefore lack factual accuracy\. RWKV alone, however, also gives the wrong answer about water and drifts after the primary colors, and a causal relation between the conversion error atL=4L\{=\}4and individual wrong answers has not been verified\.

## VDiscussion

In this experiment, a shared representation withρctr=0\.901\\rho\_\{\\mathrm\{ctr\}\}\\\!=\\\!0\.901could be learned between corresponding layers of frozen models from different families\. The range in which this conclusion applies, however, is narrow\. The correspondence was measured under the favorable conditions of the same tokenizer, hidden width, depth, and input, in a domain close to Alpaca\. The shuffle control rules out a token\-independent match of distributions, but we have not separated whether the observed correspondence arises from meaning, vocabulary, position, or syntax\. Performance also decreases sharply on WikiText, outside the training domain \(§[IV\-F](https://arxiv.org/html/2609.38261#S4.SS6)\)\. This study therefore claims only the existence of a shared representation that corresponds token by token under favorable conditions, not a general common semantic space\.

A successful functional stitch does not mean that the two models represent the same information\. Smith et al\.\[[33](https://arxiv.org/html/2609.38261#bib.bib32)\]showed that stitching can succeed between different tasks, between images and bird\-song audio, and even from randomly generated, clustered pseudo\-representations\. The stitching layers of their discriminative models, however, were trained with the task loss, the scheme that Balogh and Jelasity\[[34](https://arxiv.org/html/2609.38261#bib.bib33)\]point out rates distant layers as similar by producing out\-of\-distribution representations\. Our adapter, which uses no task loss and is trained with self\-reconstruction and latent alignment, is closer to the direct matching of representations that the same paper finds to balance structural and functional requirements\. Smith et al\. nonetheless obtained apparent alignment even with low\-capacity stitching layers and with autoencoder stitches trained by matching latent codes, and our adapter has higher capacity\. Our main reason for judging that the observed alignment cannot be explained by the capacity of the adapter alone is that a low\-capacity affine map already achieves substantial cross\-family prediction \(§[IV\-C](https://arxiv.org/html/2609.38261#S4.SS3)\)\.

A high latent correlation also does not guarantee high read\-out accuracy, so alignment and decodability must be evaluated separately, and other work further distinguishes decodability from use by the receiving model\. The preprint of Zhang and Xin\[[35](https://arxiv.org/html/2609.38261#bib.bib31)\]showed that a linear map that transfers activations between small models of the Pythia series achievesR2=0\.883R^\{2\}=0\.883in a normalized space, yet replacing the hidden states of the receiving model with the converted activations lowers performance on tasks that require several reasoning steps, even after rescaling to the receiver’s norm, and weak additive injection does not improve it\. They concluded that the training objective should optimize how the receiving model uses the activations; this point also applies to our adapter, whose objective does not include the receiving model’s output\. The unstandardization in Equation \([2](https://arxiv.org/html/2609.38261#S3.E2)\) differs from their norm rescaling in restoring the per\-dimension mean and scale, but we have not isolated the contribution of this difference\.

## VILimitations of This Study

Some of the evaluation questions may be contained in the training data of a parent model\. Pythia was instruction\-tuned on a subset of FLAN V2, and the tasks that make up FLAN V2 include ARC and SciQ\. Matching the corresponding part of the released mixture verbatim finds1919ARC questions and44SciQ questions among the evaluation questions\. The matching covers only verbatim matches and cannot detect paraphrases or formatting differences\. Questions shorter than66words, which we excluded to avoid chance matches, were not searched either,4040for ARC and1111for SciQ\. The detected counts are therefore lower bounds\. The matching also covers only the FLAN V2 and Chain\-of\-Thought parts of Pythia’s mixture, not RWKV’s instruction data or The Pile, on which both models were pretrained\. Recomputing the accuracy of every configuration from the per\-question correctness records without these questions changes it by at most0\.220\.22points and does not change the conclusion of any comparison discussed in the text\. On these questions, moreover, the accuracy of RWKV, whose listed instruction data do not include FLAN V2, changes relative to the remaining questions in the same direction as that of Pythia, by−9\.2\-9\.2points on ARC and\+8\.0\+8\.0points on SciQ, so this pattern is also consistent with an explanation based on the difficulty of the questions themselves\. ARC and SciQ nonetheless remain kinds of tasks that Pythia has already learned, so the values in this paper should be read as measuring the retention of ability after composition rather than the acquisition of new ability\.

The adapter was trained with a single random seed, and although22tasks×2\\times 2directions×4\\times 4switch layers and several controls were evaluated, no correction for multiple comparisons was applied\. The main read\-out accuracies were also computed on evaluation data that were used to adjust the learning rate and to select the checkpoint, and may be somewhat optimistic\. The same caution applies to the values in §[IV\-C](https://arxiv.org/html/2609.38261#S4.SS3), which use sequences overlapping these evaluation data\. The intervention on the nonlinear part was performed on a single checkpoint, and whether it generalizes to adapters trained under different conditions is unknown\. Each composition also has a single switch layer, and how conversion error accumulates over several switch layers has not been measured\. Furthermore, the multiple\-choice evaluation is based on likelihood and does not directly measure the ability to generate the correct answer, and the four generation examples in each direction are qualitative supplements, not a comprehensive evaluation of generation ability\.

The long\-context cross\-entropy was measured up to context length20482048to match the maximum number of positions in Pythia’s configuration\. Measurement points beyond the10241024specified by the converted RWKV configuration lie outside the range the configuration assumes, and, as noted in §[III](https://arxiv.org/html/2609.38261#S3), the context length used to train the original checkpoint cannot be confirmed\. This evaluation also measures language\-modeling quality and does not directly measure reading comprehension that requires long\-range dependencies\. Ablations overσ\\sigma,λ\\lambda, the latent width, the number of residual blocks, and the amount of training data, as well as an alignment control with two models of the same family, were not carried out in this study and are left for future work\.

## VIIConclusion

We built NinaXander, a series of chimera models that connect frozen blocks from different architecture families at a single point through one shared\-latent adapter, and tested with RWKV and Pythia whether frozen models from different families can be recombined post hoc\. Under favorable conditions the alignment between corresponding layers is learnable, and the88chimera models over22directions×\\times44switch layers, obtained from one adapter without retraining, answered multiple\-choice questions, and the22configurations atL=4L\{=\}4produced syntactically well\-formed text\. Pythia→\\toRWKV \(L=4L\{=\}4\) reduced the Transformer KV cache by84\.4%84\.4\\%with accuracy not significantly different from that of RWKV alone, but it did not exceed Pythia\. The configurations that exceed a Pythia early exit with the same KV cache size by a wide margin on both tasks, however, are those that reduce the Transformer layers to1717or fewer\. In all layers, each cross\-family read\-out was also lower than the self\-reconstruction that uses the same decoder, and performance decreased sharply on WikiText, outside the training domain\. The significance of NinaXander therefore lies not in a higher\-performing new model but in showing a way to recombine frozen models post hoc\. The contribution of this study is to show that this recombination is partly possible and can create new trade\-off points between frozen parent models, and to describe, on the basis of measurements, the possibilities and limits of composition in one favorable case\. Future work includes generalization through training across several domains, identifying what limits the self\-reconstruction accuracy, selecting complementary parent models, measuring how conversion error accumulates over several switch layers, and reproduction with multiple random seeds\.

## Acknowledgment

This work was supported by the JST Next\-Generation Edge AI Semiconductor Research and Development Program, Grant Number JPMJES2511\. This work was also supported by the Joint Usage/Research Center for Interdisciplinary Large\-scale Information Infrastructures \(JHPCN\) in Japan \(Project ID: jh260017\)\.

## References

- \[1\]K\. Lenc and A\. Vedaldi\(2019\)Understanding image representations by measuring their equivariance and equivalence\.Int\. J\. Comput\. Vision127\(5\),pp\. 456–476\.External Links:ISSN 0920\-5691,[Link](https://doi.org/10.1007/s11263-018-1098-y),[Document](https://dx.doi.org/10.1007/s11263-018-1098-y)Cited by:[§II\-B](https://arxiv.org/html/2609.38261#S2.SS2.p1.1)\.
- \[2\]Y\. Bansal, P\. Nakkiran, and B\. Barak\(2021\)Revisiting model stitching to compare neural representations\.InProceedings of the 35th International Conference on Neural Information Processing Systems,NIPS ’21,Red Hook, NY, USA\.External Links:ISBN 9781713845393Cited by:[§II\-B](https://arxiv.org/html/2609.38261#S2.SS2.p1.1)\.
- \[3\]A\. Csiszárik, P\. Kőrösi\-Szabó, Á\. Matszangosz, G\. Papp, and D\. Varga\(2021\)Similarity and matching of neural network representations\.InAdvances in Neural Information Processing Systems,M\. Ranzato, A\. Beygelzimer, Y\. Dauphin, P\.S\. Liang, and J\. W\. Vaughan \(Eds\.\),Vol\.34,pp\. 5656–5668\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2021/file/2cb274e6ce940f47beb8011d8ecb1462-Paper.pdf)Cited by:[§II\-B](https://arxiv.org/html/2609.38261#S2.SS2.p1.1)\.
- \[4\]B\. Hu, S\. Li, S\. Agarwal, M\. Lee, A\. Jajoo, J\. Li, L\. Xu, G\. Kim, D\. Kim, H\. Xu, A\. Zhang, and A\. Akella\(2025\)StitchLLM: serving LLMs, one block at a time\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 26887–26903\.External Links:[Link](https://aclanthology.org/2025.acl-long.1305/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1305),ISBN 979\-8\-89176\-251\-0Cited by:[§II\-C](https://arxiv.org/html/2609.38261#S2.SS3.p1.1)\.
- \[5\]Q\. Zhang, P\. Bhargava, C\. Bi, C\. X\. Cai, J\. N\. Foerster, J\. Fu, P\. S\. Koura, R\. Silva, S\. Shen, E\. Dinan, S\. Gururangan, and M\. Lewis\(2025\)BTS: harmonizing specialized experts into a generalist LLM\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 6816–6834\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.347/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.347),ISBN 979\-8\-89176\-332\-6Cited by:[§II\-C](https://arxiv.org/html/2609.38261#S2.SS3.p1.1)\.
- \[6\]R\. Bansal, B\. Samanta, S\. Dalmia, N\. Gupta, S\. Ganapathy, A\. Bapna, P\. Jain, and P\. Talukdar\(2024\)LLM augmented LLMs: expanding capabilities through composition\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=jjA4O1vJRz)Cited by:[§II\-C](https://arxiv.org/html/2609.38261#S2.SS3.p1.1)\.
- \[7\]N\. Roberts, S\. Guo, Z\. Gao, S\. S\. S\. N\. GNVV, S\. Cromp, C\. Wu, C\. Duan, and F\. Sala\(2025\)Pretrained hybrids with MAD skills\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=8xSbwT3763)Cited by:[§II\-C](https://arxiv.org/html/2609.38261#S2.SS3.p1.1)\.
- \[8\]L\. Moschella, V\. Maiorca, M\. Fumero, A\. Norelli, F\. Locatello, and E\. Rodolà\(2023\)Relative representations enable zero\-shot latent space communication\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SrC-nwieGJ)Cited by:[§II\-C](https://arxiv.org/html/2609.38261#S2.SS3.p2.1)\.
- \[9\]V\. Maiorca, L\. Moschella, A\. Norelli, M\. Fumero, F\. Locatello, and E\. Rodolà\(2023\)Latent space translation via semantic alignment\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§II\-C](https://arxiv.org/html/2609.38261#S2.SS3.p2.1)\.
- \[10\]S\. Grant\(2025\)Model alignment search\.InSecond Workshop on Representational Alignment at ICLR 2025,External Links:[Link](https://openreview.net/forum?id=8uNenpoGlh)Cited by:[§II\-C](https://arxiv.org/html/2609.38261#S2.SS3.p3.1)\.
- \[11\]A\. Chen, J\. Merullo, A\. Stolfo, and E\. Pavlick\(2025\)Transferring linear features across language models with model stitching\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=Qvvy0X63Fv)Cited by:[§II\-C](https://arxiv.org/html/2609.38261#S2.SS3.p3.1),[§II\-F](https://arxiv.org/html/2609.38261#S2.SS6.p1.1)\.
- \[12\]C\. Flamant, U\. Ghai, and K\. Shimizu\(2026\)The bicameral model: bidirectional hidden\-state coupling between parallel language models\.Note:arXiv:2605\.11167,[https://arxiv\.org/abs/2605\.11167](https://arxiv.org/abs/2605.11167)External Links:2605\.11167,[Link](https://arxiv.org/abs/2605.11167)Cited by:[§II\-C](https://arxiv.org/html/2609.38261#S2.SS3.p4.1)\.
- \[13\]D\. Goldstein, F\. Obeid, E\. Alcaide, G\. Song, and E\. Cheah\(2024\)GoldFinch: high performance rwkv/transformer hybrid with linear pre\-fill and extreme kv\-cache compression\.Note:arXiv:2407\.12077,[https://arxiv\.org/abs/2407\.12077](https://arxiv.org/abs/2407.12077)External Links:2407\.12077,[Link](https://arxiv.org/abs/2407.12077)Cited by:[§II\-C](https://arxiv.org/html/2609.38261#S2.SS3.p4.1)\.
- \[14\]N\. F\. Oozeer, D\. Nathawani, N\. Prakash, M\. Lan, A\. HARRASSE, and A\. Abdullah\(2025\)Activation space interventions can be transferred between large language models\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=HXOicJsmMQ)Cited by:[§II\-C](https://arxiv.org/html/2609.38261#S2.SS3.p4.1)\.
- \[15\]S\. Kornblith, M\. Norouzi, H\. Lee, and G\. Hinton\(2019\)Similarity of neural network representations revisited\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 3519–3529\.External Links:[Link](https://proceedings.mlr.press/v97/kornblith19a.html)Cited by:[§II\-D](https://arxiv.org/html/2609.38261#S2.SS4.p1.1)\.
- \[16\]M\. Raghu, J\. Gilmer, J\. Yosinski, and J\. Sohl\-Dickstein\(2017\)SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability\.InProceedings of the 31st International Conference on Neural Information Processing Systems,NIPS’17,Red Hook, NY, USA,pp\. 6078–6087\.External Links:ISBN 9781510860964Cited by:[§II\-D](https://arxiv.org/html/2609.38261#S2.SS4.p1.1)\.
- \[17\]A\. S\. Morcos, M\. Raghu, and S\. Bengio\(2018\)Insights on representational similarity in neural networks with canonical correlation\.InProceedings of the 32nd International Conference on Neural Information Processing Systems,NIPS’18,Red Hook, NY, USA,pp\. 5732–5741\.Cited by:[§II\-D](https://arxiv.org/html/2609.38261#S2.SS4.p1.1)\.
- \[18\]J\. Wang, X\. Ge, W\. Shu, Q\. Tang, Y\. Zhou, Z\. He, and X\. Qiu\(2025\)Towards universality: studying mechanistic similarity across language model architectures\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=2J18i8T0oI)Cited by:[§II\-D](https://arxiv.org/html/2609.38261#S2.SS4.p1.1)\.
- \[19\]M\. Wortsman, G\. Ilharco, S\. Y\. Gadre, R\. Roelofs, R\. Gontijo\-Lopes, A\. S\. Morcos, H\. Namkoong, A\. Farhadi, Y\. Carmon, S\. Kornblith, and L\. Schmidt\(2022\)Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.InProceedings of the 39th International Conference on Machine Learning,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvari, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.162,pp\. 23965–23998\.External Links:[Link](https://proceedings.mlr.press/v162/wortsman22a.html)Cited by:[§II\-E](https://arxiv.org/html/2609.38261#S2.SS5.p1.1)\.
- \[20\]P\. Yadav, D\. Tam, L\. Choshen, C\. Raffel, and M\. Bansal\(2023\)TIES\-merging: resolving interference when merging models\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§II\-E](https://arxiv.org/html/2609.38261#S2.SS5.p1.1)\.
- \[21\]T\. Akiba, M\. Shing, Y\. Tang, Q\. Sun, and D\. Ha\(2025\)Evolutionary optimization of model merging recipes\.Nature Machine Intelligence7\(2\),pp\. 195–204\.External Links:[Document](https://dx.doi.org/10.1038/s42256-024-00975-8),[Link](https://doi.org/10.1038/s42256-024-00975-8),ISSN 2522\-5839Cited by:[§II\-E](https://arxiv.org/html/2609.38261#S2.SS5.p1.1)\.
- \[22\]F\. Wan, X\. Huang, D\. Cai, X\. Quan, W\. Bi, and S\. Shi\(2024\)Knowledge fusion of large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=jiDsk12qcz)Cited by:[§II\-E](https://arxiv.org/html/2609.38261#S2.SS5.p1.1)\.
- \[23\]L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. Leahy\(2020\)The pile: an 800gb dataset of diverse text for language modeling\.Note:arXiv:2101\.00027,[https://arxiv\.org/abs/2101\.00027](https://arxiv.org/abs/2101.00027)External Links:2101\.00027,[Link](https://arxiv.org/abs/2101.00027)Cited by:[§III\-A](https://arxiv.org/html/2609.38261#S3.SS1.p2.1)\.
- \[24\]B\. Peng, E\. Alcaide, Q\. Anthony, A\. Albalak, S\. Arcadinho, S\. Biderman, H\. Cao, X\. Cheng, M\. Chung, L\. Derczynski, X\. Du, M\. Grella, K\. Gv, X\. He, H\. Hou, P\. Kazienko, J\. Kocon, J\. Kong, B\. Koptyra, H\. Lau, J\. Lin, K\. S\. I\. Mantri, F\. Mom, A\. Saito, G\. Song, X\. Tang, J\. Wind, S\. Woźniak, Z\. Zhang, Q\. Zhou, J\. Zhu, and R\. Zhu\(2023\)RWKV: reinventing RNNs for the transformer era\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 14048–14077\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.936/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.936)Cited by:[§III\-A](https://arxiv.org/html/2609.38261#S3.SS1.p3.1)\.
- \[25\]R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto\(2023\)Stanford alpaca: an instruction\-following llama model\.GitHub\.Note:[https://github\.com/tatsu\-lab/stanford\_alpaca](https://github.com/tatsu-lab/stanford_alpaca)Cited by:[§III\-A](https://arxiv.org/html/2609.38261#S3.SS1.p3.1)\.
- \[26\]S\. Chaudhary\(2023\)Code alpaca: an instruction\-following llama model trained on code generation instructions\.GitHub\.Note:[https://github\.com/sahil280114/codealpaca](https://github.com/sahil280114/codealpaca)Cited by:[§III\-A](https://arxiv.org/html/2609.38261#S3.SS1.p3.1)\.
- \[27\]LMSYS Org\(2023\)FastChat: an open platform for training, serving, and evaluating large language model based chatbots\.GitHub\.Note:[https://github\.com/lm\-sys/FastChat](https://github.com/lm-sys/FastChat)Cited by:[§III\-A](https://arxiv.org/html/2609.38261#S3.SS1.p3.1)\.
- \[28\]S\. Biderman, H\. Schoelkopf, Q\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff, A\. Skowron, L\. Sutawika, and O\. Van Der Wal\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.InProceedings of the 40th International Conference on Machine Learning,ICML’23\.Cited by:[§III\-A](https://arxiv.org/html/2609.38261#S3.SS1.p4.1)\.
- \[29\]Y\. Wang, H\. Ivison, P\. Dasigi, J\. Hessel, T\. Khot, K\. R\. Chandu, D\. Wadden, K\. MacMillan, N\. A\. Smith, I\. Beltagy, and H\. Hajishirzi\(2023\)How far can camels go? exploring the state of instruction tuning on open resources\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§III\-A](https://arxiv.org/html/2609.38261#S3.SS1.p4.1)\.
- \[30\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.Note:arXiv:1803\.05457,[https://arxiv\.org/abs/1803\.05457](https://arxiv.org/abs/1803.05457)External Links:1803\.05457,[Link](https://arxiv.org/abs/1803.05457)Cited by:[§III\-E](https://arxiv.org/html/2609.38261#S3.SS5.p4.1)\.
- \[31\]J\. Welbl, N\. F\. Liu, and M\. Gardner\(2017\)Crowdsourcing multiple choice science questions\.InProceedings of the 3rd Workshop on Noisy User\-generated Text,L\. Derczynski, W\. Xu, A\. Ritter, and T\. Baldwin \(Eds\.\),Copenhagen, Denmark,pp\. 94–106\.External Links:[Link](https://aclanthology.org/W17-4413/),[Document](https://dx.doi.org/10.18653/v1/W17-4413)Cited by:[§III\-E](https://arxiv.org/html/2609.38261#S3.SS5.p4.1)\.
- \[32\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\(2017\)Pointer sentinel mixture models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Byj72udxe)Cited by:[§III\-E](https://arxiv.org/html/2609.38261#S3.SS5.p6.1)\.
- \[33\]D\. Smith, H\. Mannering, and A\. Marcu\(2025\)Functional alignment can mislead: examining model stitching\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 55972–55998\.External Links:[Link](https://proceedings.mlr.press/v267/smith25a.html)Cited by:[§V](https://arxiv.org/html/2609.38261#S5.p2.1)\.
- \[34\]A\. Balogh and M\. Jelasity\(2025\)How not to stitch representations to measure similarity: task loss matching versus direct matching\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 15472–15480\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v39i15.33698)Cited by:[§V](https://arxiv.org/html/2609.38261#S5.p2.1)\.
- \[35\]P\. Zhang and J\. Xin\(2026\)A negative result on cross\-model activation transfer in a pythia multi\-hop setting\.Note:arXiv:2606\.03280,[https://arxiv\.org/abs/2606\.03280](https://arxiv.org/abs/2606.03280)External Links:2606\.03280,[Link](https://arxiv.org/abs/2606.03280)Cited by:[§V](https://arxiv.org/html/2609.38261#S5.p3.1)\.

## Appendix AReproduction Information

- •Models: the two parent models of §[III](https://arxiv.org/html/2609.38261#S3)are used in FP16\. The released artifactparent\_fingerprints\.jsonrecords the SHA\-256 \(Secure Hash Algorithm 256\-bit\) hashes of the local parent models used for training\.
- •Adapter training: the adapter was trained on44NVIDIA V100 GPUs\.
- •Data: the evaluation sequences were taken from Alpaca from character position2,000,0002\{,\}000\{,\}000onward, which does not depend on the number of training tokens\. The standardization statistics are computed from the training residuals of each layer and are also used for unstandardization\.
- •Evaluation: for the RWKV back model of Pythia→\\toRWKV, options are always evaluated with a batch size of11\.
- •Code and structured data: the repository given in §[III](https://arxiv.org/html/2609.38261#S3)contains the experiment code, the raw data of each evaluation with aggregate tables of the reported values, the reproduction procedures, and a check script that automatically matches the numbers in the text against the aggregate tables\.
- •Hugging Face releases: the inference artifacts as of July 30, 2026 are stored in the following three model repositories\. - – - – - – The latter two bundle the two frozen parent models in FP16, the shared tokenizer, the adapter, and custom inference code that selects the switch layer fromL=4,8,16,24L\{=\}4,8,16,24, so each runs as a standalone chimera model\. Each is fixed to one connection direction, and its default switch layerL=4L\{=\}4was chosen after seeing the evaluation results, as in §[IV\-G](https://arxiv.org/html/2609.38261#S4.SS7)\. No independent data split was used for this selection\. All three repositories contain model cards in English, Japanese, and Simplified Chinese, a NOTICE file that records the attribution of the parent models, weights in the Safetensors format, and a file list with SHA\-256 hashes\. The license is the AI2 AI Model License for non\-commercial research, matching Pythia, the more restrictive parent model\.