WARP: Weight-Space Analysis for Recovering Training Data Portfolios

# WARP: Weight-Space Analysis for Recovering Training Data Portfolios
Source: [https://arxiv.org/html/2607.01686](https://arxiv.org/html/2607.01686)
Aditya GoyalJohn CooperUniversity of Wisconsin\-MadisonFrederic SalaUniversity of Wisconsin\-Madison

###### Abstract

Foundation models are routinely released to the public, yet the data recipes used to train them—such as domain mixture weights that determine how different sources are sampled—are rarely disclosed\. This creates an access asymmetry: researchers study the resulting models but lack visibility into the training distribution that produces them\. Prior works for inferring training data, such as membership inference, detect at the level of individual samples and thus cannot characterize the global composition of the training corpus\. We introduce WARP, a framework that recovers a fine\-tuned model’s training mixtures directly from its released weights\. WARP interpolates between the base and fine\-tuned models using model merging, generating*pseudo\-checkpoints*that approximate the missing training trajectory and expose a geometric footprint of the training data in the weight space\. From these simulated footprints, WARP extracts geometric features and maps them to domain proportions using either a parameter\-free softmax readout or an MLP projector trained on synthetic mixtures\. In controlled experiments with BERT and GPT\-2, WARP recovers domain mixtures with an average MAE as low as0\.0460\.046and0\.1040\.104respectively, outperforming membership inference and a variant with access to the true training trajectory\.

††\*Equal Contribution\. Corresponding Authors: ¡thuang273, agoyal33¿@wisc\.edu11footnotetext:This work appears in the ICML 2026 Workshop on Weight\-Space Symmetries \(WSS\): from Foundations to Practical Applications\.22footnotetext:Our source code is available[here](https://github.com/SprocketLab/WARP)\.## 1Introduction

Foundation models are trained on massive, heterogeneous corpora spanning web text, source code, and scientific literature\. One primary determinant of a model’s ultimate capability is its*domain mixture*: the proportions in which samples are drawn from these diverse sources\[[6](https://arxiv.org/html/2607.01686#bib.bib22)\]\. Recent studies have shown that tuning this data mixture is not merely a matter of ensuring knowledge variety, but of training efficiency—a well\-designed mixture can yield models that substantially outperform naive sampling while requiring fewer optimization steps\[[31](https://arxiv.org/html/2607.01686#bib.bib19);[11](https://arxiv.org/html/2607.01686#bib.bib20);[4](https://arxiv.org/html/2607.01686#bib.bib37);[9](https://arxiv.org/html/2607.01686#bib.bib21)\]\.

Yet the mixing recipes behind most frontier models remain proprietary and opaque\. This secrecy reflects a broader*access asymmetry*in modern AI: general\-purpose models are increasingly released to the public, while the carefully curated datasets used to produce them are not\. Such opacity creates structural barriers for the research community\. Practitioners who continually fine\-tune a released model without knowing its original distribution risk unintended capability drift, and more broadly, recovering a model’s data portfolio is useful for auditing data contamination\[[14](https://arxiv.org/html/2607.01686#bib.bib35);[13](https://arxiv.org/html/2607.01686#bib.bib36);[21](https://arxiv.org/html/2607.01686#bib.bib33)\]and for interpreting divergent model behaviors\[[28](https://arxiv.org/html/2607.01686#bib.bib32)\]\.

Breaking through this barrier, however, is far from straightforward\. Existing approaches to domain mixture optimization operate primarily in the*forward*direction—*from data to model*\. They either establish an optimized portfolio for a given dataset via gradient information\[[11](https://arxiv.org/html/2607.01686#bib.bib20);[9](https://arxiv.org/html/2607.01686#bib.bib21)\], or train multiple proxy models that capture the relationship from mixtures to downstream performance\[[29](https://arxiv.org/html/2607.01686#bib.bib23);[20](https://arxiv.org/html/2607.01686#bib.bib24);[19](https://arxiv.org/html/2607.01686#bib.bib25);[27](https://arxiv.org/html/2607.01686#bib.bib38)\]\. These natural designs make it hard to characterize already\-released weights\. The alternative—working*backward from a model to its training data*—has traditionally been framed as Membership Inference \(MI\)\[[10](https://arxiv.org/html/2607.01686#bib.bib13);[8](https://arxiv.org/html/2607.01686#bib.bib5);[23](https://arxiv.org/html/2607.01686#bib.bib14);[26](https://arxiv.org/html/2607.01686#bib.bib34)\], which asks whether a particular sample appeared in the training pool, typically by comparing its attribution \(e\.g\., loss or output logits\) against a reference distribution\. However, MI is fundamentally a sample\-level detection task and offers no holistic view of underlying domain proportions\.

![Refer to caption](https://arxiv.org/html/2607.01686v1/figures/problem_figure.png)Figure 1:Two directions of the domain mixture problem\.Left:the well\-studied*forward*problem—given a corpus, search for an optimized mixtureπ⋆\\pi^\{\\star\}over domains \(e\.g\., web, math, code, agentic data\) and train a modelθ⋆\\theta^\{\\star\}under it, whose final weights are then publicly released\.Right:the*inverse*problem we study—given only the released endpoints\(θbase⋆,θref⋆\)\(\\theta^\{\\star\}\_\{\\mathrm\{base\}\},\\theta^\{\\star\}\_\{\\mathrm\{ref\}\}\)and a small probing dataset, with the corpus and true training trajectory withheld, recover an estimateπ^\\hat\{\\pi\}of the domain proportions that produced the fine\-tuned model\.One promising direction has recently appeared: using*weight\-space geometry*, the one artifact always available, to recover insights about training data\[[24](https://arxiv.org/html/2607.01686#bib.bib4);[3](https://arxiv.org/html/2607.01686#bib.bib3);[16](https://arxiv.org/html/2607.01686#bib.bib31)\]\. For example, the Mimic Score\[[16](https://arxiv.org/html/2607.01686#bib.bib31)\]combines a directional vector in weight\-space with training gradients to estimate sample utility for data curation\. These methods share a simple premise:*a model’s parameters encode traces of its training history, and therefore of the data that produced it*\. Yet they typically require access to*intermediate*training checkpoints to reconstruct the learning trajectory—a luxury rarely available in practice\. For the vast majority of released models, only the final weights \(e\.g\.,Instruct, Chatversions\) and the base model \(e\.g\., pretrained\-only\) can be obtained: the origin and the destination, with no record of the path between them\.

In this work, we break this asymmetry withWARP\(Weight\-spaceAnalysis forRecovering Training DataPortfolios\), a framework that recovers a fine\-tuned model’s domain mixtures directly from its released weights\. We argue that*though the true training trajectory is lost, alternative interpolants between the two endpoints still expose usable structure in the surrounding weight\-space geometry\.*Specifically, we use model merging techniques to interpolate between the base and fine\-tuned models, generating*pseudo\-checkpoints*that stand in for the unobserved parameter evolution\. From these simulated trajectories, their extracted geometric features allow us to capture a distributional footprint of the training data\. Given a small probing dataset,\(i\)a parameter\-free softmax readout can recover a coarse mixture estimate; or\(ii\)a stronger variant uses synthetic mixtures to build an MLP for learning the projection from geometric features to predicted mixture\.

We empirically validate WARP in a controlled setting by training BERT\[[7](https://arxiv.org/html/2607.01686#bib.bib1)\]and GPT\-2\[[25](https://arxiv.org/html/2607.01686#bib.bib15)\]under known mixtures and recovering the proportions from each fine\-tuned checkpoint\. Across forty experimental trials, WARP achieves a mean absolute error \(MAE\) as low as0\.0460\.046on BERT and0\.1040\.104on GPT\-2 averaged across four text datasets, outperforming sample\-level MI baselines and a variant that allows using the true training trajectory\. Moreover, the WARP estimator remains accurate on early\-stop, converged, and overtrained checkpoints—mirroring today’s post\-training regime, where released models are routinely pushed beyond compute\-optimal budgets, demonstrating its robustness to different training recipes\.

We summarize our contributions as follows:

- •Closing the Access Asymmetry\.We formalize the recovery of a fine\-tuned model’s domain mixture from its weight\-space footprints, relying only on artifacts available for most of the released models\.
- •Simulated Training Trajectories\.We propose WARP, which sidesteps the need for intermediate checkpoints by reconstructing the missing path through model merging, then projects the resulting weight\-space geometry to domain mixture estimates\.
- •Strong Empirical Performance\.Across controlled fine\-tuning runs, WARP outperforms sample\-level MI baselines and even a variant that has access to the true training trajectory\.
- •Robustness Across Training Recipes\.WARP remains accurate when recovering converged and overtrained checkpoints, demonstrating robustness to diverse training recipes\.

## 2Related Work

Our work sits at the intersection of three research threads:\(i\)domain mixture optimization,\(ii\)training data inference, and\(iii\)model merging techniques\.

#### Domain Mixture Optimization\.

Establishing an ideal data mixture plays a critical role in training foundation models\[[6](https://arxiv.org/html/2607.01686#bib.bib22)\]\. Rather than relying on grid search to discover the best configuration, a recent line of work has developed principled methods for optimizing the data mixture\[[31](https://arxiv.org/html/2607.01686#bib.bib19);[4](https://arxiv.org/html/2607.01686#bib.bib37);[5](https://arxiv.org/html/2607.01686#bib.bib18)\]\. These methods broadly fall into two groups: \(i\) static mixing and \(ii\) dynamic reweighting\. Static methods fix the mixture before training, typically using proxy models to predict how mixture weights affect downstream performance—for example, by fitting a regression model that maps weights to performance\[[29](https://arxiv.org/html/2607.01686#bib.bib23);[20](https://arxiv.org/html/2607.01686#bib.bib24);[19](https://arxiv.org/html/2607.01686#bib.bib25);[27](https://arxiv.org/html/2607.01686#bib.bib38)\], or by leveraging gradient information from a held\-out set to optimize the weights\[[9](https://arxiv.org/html/2607.01686#bib.bib21)\]\. Dynamic methods instead adapt the mixture weights throughout training, for example by constructing a gradient\-alignment matrix \(R&B\)\[[11](https://arxiv.org/html/2607.01686#bib.bib20)\]or by casting data mixing as a multi\-armed bandit problem\[[2](https://arxiv.org/html/2607.01686#bib.bib29)\]\.*All of these methods operate in the forward direction: given the data, they find the best mixture\.*In contrast, we address the reverse problem: given a model, we aim to recover the mixture that produced it\.

#### Training Data Inference\.

Inferring properties of a model’s training data, especially for large language models, is of growing interest, motivated both by privacy concerns—over the unauthorized use of copyrighted and personal information\[[14](https://arxiv.org/html/2607.01686#bib.bib35);[22](https://arxiv.org/html/2607.01686#bib.bib28)\]—and by the practical utility of extracting high\-quality data insights for next\-round data curation\[[16](https://arxiv.org/html/2607.01686#bib.bib31)\]\. A common formulation is*membership inference*\[[10](https://arxiv.org/html/2607.01686#bib.bib13);[8](https://arxiv.org/html/2607.01686#bib.bib5);[23](https://arxiv.org/html/2607.01686#bib.bib14);[26](https://arxiv.org/html/2607.01686#bib.bib34)\], which detects whether a particular sample was used in training by comparing the model’s behavior against a reference distribution, for instance, by examining the difference in output logits between an untuned and a fine\-tuned model\. However, membership inference operates at the level of individual samples and therefore lacks a holistic view of how domains interact\. Even if the detected samples are aggregated to recover domain mixtures, the resulting estimate is confined to the probing dataset and remains suboptimal\. In this work, we propose a framework that*recovers a model’s training trajectory in weight\-space and leverages domain\-level geometric features to recover the training mixture\.*

#### Model Merging Techniques\.

Model merging combines two or more models into a single fused model that inherits the capabilities of each\[[34](https://arxiv.org/html/2607.01686#bib.bib7);[12](https://arxiv.org/html/2607.01686#bib.bib27);[17](https://arxiv.org/html/2607.01686#bib.bib11)\]\. It operates in weight space, where model weights can be permuted\[[1](https://arxiv.org/html/2607.01686#bib.bib6);[18](https://arxiv.org/html/2607.01686#bib.bib12)\], interpolated\[[32](https://arxiv.org/html/2607.01686#bib.bib16);[36](https://arxiv.org/html/2607.01686#bib.bib9)\], stitched\[[15](https://arxiv.org/html/2607.01686#bib.bib8)\], or averaged\[[30](https://arxiv.org/html/2607.01686#bib.bib10)\]to yield versatile skills\. Most work focuses on resolving conflicts between models to maximize the fused model’s performance\. WARP departs from this view: rather than treating merging as a way to combine capabilities,*we use merging as a mechanism for simulating an unobserved training trajectory—constructing pseudo\-checkpoints that stand in for the missing path between a base model and its fine\-tuned descendant*\.

## 3Framework

We proposeWARPto recover the domain mixture of a fine\-tuned model under two practical constraints: \(i\) the final weights and the corresponding base model are available, and \(ii\) the practitioner has access to a data sourceSSfrom which the fine\-tuning data is drawn \(e\.g\., the web datasets\), and can freely sample a small set of labeled examples from it\. We argue that*the geometry traced between a base model and its fine\-tuned descendant encodes a distributional footprint of the training data, and this footprint can be read out by measuring how samples from each domain align with the geometry*\.

![Refer to caption](https://arxiv.org/html/2607.01686v1/figures/image.png)Figure 2:Overview of WARP\.\(1\) Simulate the Training Trajectory:since the true fine\-tuning path is unobserved, we first approximate it with a sequence of pseudo\-checkpoints obtained via model merging\.\(2\) Distill Geometric Footprint:at each pseudo\-checkpoint, we compute the Mimic Score by projecting the gradient of probing samples from each domain onto the direction pointing to reference model\. This measures how strongly each domain aligns with the path actually taken during fine\-tuning\.\(3\) Project from Geometry to Mixture:the resulting footprint matrix is mapped to a predicted domain mixture, either via an unsupervised softmax readout or a supervised MLP projector trained on synthetic mixture\-footprint pairs\.We begin with notation, then describe how we bypass the unobserved learning trajectory through simulation \(§[3\.1](https://arxiv.org/html/2607.01686#S3.SS1)\), how the resulting weight\-space geometry is converted into a domain\-level signal \(§[3\.2](https://arxiv.org/html/2607.01686#S3.SS2)\), and finally how this signal is mapped to a predicted domain mixture \(§[3\.3](https://arxiv.org/html/2607.01686#S3.SS3)\)\. We demonstrate our framework workflow in Figure[2](https://arxiv.org/html/2607.01686#S3.F2)\. Moreover, we provide a unified algorithm in Appendix[A](https://arxiv.org/html/2607.01686#A1)\.

#### Notation\.

Letθbase\\theta\_\{\\text\{base\}\}denote the parameters of a publicly available base model \(pretrained\-only\), andθref\\theta\_\{\\text\{ref\}\}those of a released fine\-tuned model derived from it\. Its fine\-tuning dataDrefD\_\{\\text\{ref\}\}is sampled from a sourceSSthat spansKKdomains, with unknown mixture proportionsπ⋆=\(π1⋆,…,πK⋆\)∈ΔK−1\\pi^\{\\star\}=\(\\pi^\{\\star\}\_\{1\},\\dots,\\pi^\{\\star\}\_\{K\}\)\\in\\Delta^\{K\-1\}, whereΔK−1\\Delta^\{K\-1\}is the probability simplex\. Our goal is to estimateπ⋆\\pi^\{\\star\}fromθbase\\theta\_\{\\text\{base\}\}andθref\\theta\_\{\\text\{ref\}\}alone, without access toDrefD\_\{\\text\{ref\}\}or any intermediate training checkpoint\. To reach this estimation, we assume the ability to sample examples fromSSin our own way, allowing us to construct a small probing datasetDprobe=⋃k=1KDkD\_\{\\text\{probe\}\}=\\bigcup\_\{k=1\}^\{K\}D\_\{k\}, where eachDk=\{\(xi,yi\)\}i=1nkD\_\{k\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\_\{k\}\}contains samples with known domain labelkkand\|Dprobe\|≪\|Dref\|\|D\_\{\\text\{probe\}\}\|\\ll\|D\_\{\\text\{ref\}\}\|\. Crucially,DprobeD\_\{\\text\{probe\}\}need not follow the same mixture asDrefD\_\{\\text\{ref\}\}—it need only cover theKKdomains of interest\. For any parameter vectorθ\\theta, we write each sample’s gradient asgi​\(θ\):=∇θℓ​\(xi,yi\)g\_\{i\}\(\\theta\):=\\nabla\_\{\\theta\}\\ell\(x\_\{i\},y\_\{i\}\)\.

### 3\.1Simulating the Training Trajectory

To exploit the full geometric information along a training path, one would ideally have access to its intermediate checkpoints\. In practice, however, the true trajectory fromθbase\\theta\_\{\\text\{base\}\}toθref\\theta\_\{\\text\{ref\}\}is rarely released, leaving us with only its two endpoints\. To recover a usable approximation, we draw on recent advances in model merging\[[33](https://arxiv.org/html/2607.01686#bib.bib2);[12](https://arxiv.org/html/2607.01686#bib.bib27)\]and interpolate between these endpoints to construct a sequence of*pseudo\-checkpoints*:θ~t:=ℳ​\(θbase,θref;t\),\\tilde\{\\theta\}\_\{t\}:=\\mathcal\{M\}\\big\(\\theta\_\{\\text\{base\}\},\\,\\theta\_\{\\text\{ref\}\};\\,t\\big\),whereℳ\\mathcal\{M\}is a merging operator\. For linear interpolation, TIES\[[32](https://arxiv.org/html/2607.01686#bib.bib16)\], and other merging methods, it is common to have\{αt\}t=1T⊂\(0,1\)\\\{\\alpha\_\{t\}\\\}\_\{t=1\}^\{T\}\\subset\(0,1\)with

θ~t\\displaystyle\\tilde\{\\theta\}\_\{t\}=\(1−αt\)​θbase\+αt​θref,t=1,…,T,\\displaystyle=\(1\-\\alpha\_\{t\}\)\\,\\theta\_\{\\text\{base\}\}\+\\alpha\_\{t\}\\,\\theta\_\{\\text\{ref\}\},\\qquad t=1,\\dots,T,as a schedule of mixing coefficients that places each pseudo\-checkpoint between the two endpoints\. Intuitively,αt\\alpha\_\{t\}plays the role of a normalized training progress: values near0yield an*early\-stage*state close toθbase\\theta\_\{\\text\{base\}\}, where the model has only begun to absorb the fine\-tuning data, while values near11yield a*late\-stage*state nearθref\\theta\_\{\\text\{ref\}\}, where most of the adaptation has already taken place\. Eachθ~t\\tilde\{\\theta\}\_\{t\}thus serves as a proxy for the parameter state at an intermediate point along the simulated trajectory\.

### 3\.2Distilling Geometric Footprints

With a simulated trajectory in hand, we now extract a domain\-level signal from pseudo\-checkpoints’ weight\-space geometry\.

#### Sample Alignment in Weight\-space\.

The Mimic Score\[[16](https://arxiv.org/html/2607.01686#bib.bib31)\]measures how well a sample’s negative gradient aligns with a direction in the weight space pointing toward a more desirable parameter set:*samples whose updates can push the model along this direction receive higher scores and are treated as higher\-value training examples\.*We adapt this quality metric to our setting by treating each pseudo\-checkpointθ~t\\tilde\{\\theta\}\_\{t\}as the current state andθref\\theta\_\{\\text\{ref\}\}as the desired target, which together define a time\-varying direction vectorvt:=θref−θ~tv\_\{t\}\\;:=\\;\\theta\_\{\\text\{ref\}\}\-\\tilde\{\\theta\}\_\{t\}\. For each probing sample\(xi,yi\)∈Dprobe\(x\_\{i\},y\_\{i\}\)\\in D\_\{\\text\{probe\}\}, the Mimic Score at stepttis the projection of its negative gradient onto the unit vector alongvtv\_\{t\}:

mi,t:=⟨−gi​\(θ~t\),vt⟩‖vt‖\.m\_\{i,t\}\\;:=\\;\\frac\{\\langle\-g\_\{i\}\(\\tilde\{\\theta\}\_\{t\}\),\\,v\_\{t\}\\rangle\}\{\\\|v\_\{t\}\\\|\}\.\(1\)Intuitively,mi,tm\_\{i,t\}reflects how strongly sampleiiwould have pulled the model along the direction actually taken during fine\-tuning\.*Samples from domains heavily represented inD*ref*D\_\{\\text\{ref\}\}should therefore yield relatively larger alignment than those from underrepresented ones*\. Since‖vt‖\\\|v\_\{t\}\\\|shrinks asαt→1\\alpha\_\{t\}\\to 1, raw scores at different steps are not directly comparable; we therefore apply min\-max normalization to\{mi,t\}i\\\{m\_\{i,t\}\\\}\_\{i\}within each steptt\.

#### Domain\-level Pooling\.

After normalization, we aggregate per\-sample derived scores into a domain\-level signal by averaging within each domainkkat each pseudo\-checkpoint:

m¯k,t:=1\|Dk\|​∑i∈Dkmi,t\.\\displaystyle\\bar\{m\}\_\{k,t\}\\;:=\\;\\frac\{1\}\{\|D\_\{k\}\|\}\\sum\_\{i\\in D\_\{k\}\}m\_\{i,t\}\.Stacking these pooled scores across allKKdomains andTTpseudo\-checkpoints yields a feature matrixM∈ℝK×TM\\in\\mathbb\{R\}^\{K\\times T\}, which serves as the*geometric footprint*of the simulated trajectory:*rowkktraces how strongly domainkkaligns with the fine\-tuning direction as the model evolves fromθ*base*\\theta\_\{\\text\{base\}\}towardθ*ref*\\theta\_\{\\text\{ref\}\}*\.

### 3\.3Mapping from Geometry to Mixture

The final step is to translate the geometric footprintMMinto a predicted mixtureπ^∈ΔK−1\\hat\{\\pi\}\\in\\Delta^\{K\-1\}\. We instantiate this mapping in two ways: using an*unsupervised*variant that reads the mixture directly offMM, or leveraging a*supervised*variant that learns the mapping from synthetic labeled pairs\.

#### Unsupervised Readout\.

The simplest mapping convertsMMinto a mixture estimate without any learnable parameters, by averaging the pooled scores across pseudo\-checkpoints and applying a softmax over domains:

π^k=exp⁡\(m˙k/τ\)∑k′=1Kexp⁡\(m˙k′/τ\),m˙k=1T​∑t=1Tm¯k,t,\\hat\{\\pi\}\_\{k\}\\;=\\;\\frac\{\\exp\(\\dot\{m\}\_\{k\}/\\tau\)\}\{\\sum\_\{k^\{\\prime\}=1\}^\{K\}\\exp\(\\dot\{m\}\_\{k^\{\\prime\}\}/\\tau\)\},\\quad\\dot\{m\}\_\{k\}\\;=\\;\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\bar\{m\}\_\{k,t\},\(2\)whereτ\>0\\tau\>0is a temperature controlling the sharpness of the predicted distribution\. This readout is*fast and label\-free*, but by collapsing the temporal dimension it cannot exploit how domain influence varies across early and late learning\.

#### Supervised Projection\.

Our second variant trains a projector using synthetic pairs constructed from the same data sourceSS\. Concretely, we drawJJrandom mixturesπ\(j\)\\pi^\{\(j\)\}from a distributionΠ\\Piover the probability simplex, designed so that‖π\(j\)−1K​𝟏‖1\\\|\\pi^\{\(j\)\}\-\\frac\{1\}\{K\}\\mathbf\{1\}\\\|\_\{1\}is uniform\. For each sampled mixture, we draw a new dataset fromSSaccordingly and fine\-tune the base model on it to obtain a synthetic reference model, then compute its footprintM\(j\)M^\{\(j\)\}using the probing datasetDprobeD\_\{\\text\{probe\}\}\. Moreover, since each fine\-tune is only a means of*generating training signal*for the projector—not a model intended for downstream use—*we do not need to optimize these mixtures*\.*That is, cheap, short fine\-tuning runs are acceptable, making synthetic pairs efficient to produce at scale*\. On these pairs, we train an MLP with a softmax output layer,fϕ:ℝK×T→ΔK−1f\_\{\\phi\}:\\mathbb\{R\}^\{K\\times T\}\\to\\Delta^\{K\-1\}, that maps the full feature matrix to a predicted mixtureπ^=fϕ​\(M\)\\hat\{\\pi\}=f\_\{\\phi\}\(M\), using a standard regression loss against the synthetic ground\-truth mixtureπ\(j\)\\pi^\{\(j\)\}\. By operating on the fullMMrather than its temporal average, this learns to weight different learning stages adaptively, capturing how domain influence evolves along the simulated trajectory\.

## 4Experiments

We evaluate WARP in a controlled setting where the ground\-truth domain mixture is known, allowing us to directly measure the recovery effectiveness\. Our goal is to verify:

1. 1\.Effectiveness of Weight\-Space Geometry\.Recovering domain mixtures from weight\-space geometry substantially outperforms random guessing and sample\-level membership inference baselines\.
2. 2\.Efficiency of Unsupervised Readout\.Even without any learned projector, a parameter\-free softmax readout can recover domain proportions accurately\.
3. 3\.Simulated Paths Suffice\.Pseudo\-checkpoints obtained via model merging can simulate the true learning path, matching—and sometimes outperforming—the variant granted access to real checkpoints\.
4. 4\.Robustness Across Training Recipes\.WARP remains accurate on both converged and overtrained checkpoints, mirroring the post\-training regime where current models are pushed beyond compute\-optimal budgets\.

#### Setups\.

We use four text datasets to evaluate WARP, spanning different domain structures: \(i\)SNLI\[[35](https://arxiv.org/html/2607.01686#bib.bib30)\]\(3 classes, natural language inference\), \(ii\)AGNews\[[37](https://arxiv.org/html/2607.01686#bib.bib26)\]\(4 classes, news topic classification\), \(iii\)Yelp\[[37](https://arxiv.org/html/2607.01686#bib.bib26)\]\(5 classes, review sentiment\), and \(iv\)Yahoo\[[37](https://arxiv.org/html/2607.01686#bib.bib26)\]\(10 classes, topic classification\)\. We treat each class as a representative domain\. For every dataset, we sample4040ground\-truth mixturesπ⋆\\pi^\{\\star\}overΔK−1\\Delta^\{K\-1\}to construct a5,0005,000\-sized dataset and fine\-tune bothBERT\-baseandGPT\-2\-Smallon data drawn according to eachπ⋆\\pi^\{\\star\}to obtainθref\\theta\_\{\\text\{ref\}\}, using the corresponding pretrained weights asθbase\\theta\_\{\\text\{base\}\}\. The probing datasetDprobeD\_\{\\text\{probe\}\}is drawn separately from the same sourceSS, and\|Dprobe\|=2,500\|D\_\{\\text\{probe\}\}\|=2\{,\}500examples sampled uniformly per domain\. Pseudo\-checkpoints are constructed atT=15T=15evenly spaced interpolation steps betweenθbase\\theta\_\{\\text\{base\}\}andθref\\theta\_\{\\text\{ref\}\}\. Recovery performance is reported using mean absolute error \(MAE\) betweenπ^\\hat\{\\pi\}andπ⋆\\pi^\{\\star\}, averaged across the4040trials\. We provide our training configurations in Appendix[B](https://arxiv.org/html/2607.01686#A2)\.

Table 1:Domain mixture recovery error \(MAE between predictedπ^\\hat\{\\pi\}and ground\-truthπ⋆\\pi^\{\\star\}, averaged over4040trials\) on BERT and GPT\-2\-Small across datasets\. Lower is better\. Best results within each model block arebolded; second\-best areunderlined\.BERTGPT\-2\-SmallMethodSNLIAGNewsYelpYahooAvg\.SNLIAGNewsYelpYahooAvg\.\(3 cls\.\)\(4 cls\.\)\(5 cls\.\)\(10 cls\.\)MAE\(3 cls\.\)\(4 cls\.\)\(5 cls\.\)\(10 cls\.\)MAEBaselinesRandom Guess0\.2940\.2940\.2390\.2390\.1920\.1920\.0900\.2040\.2040\.2940\.2940\.2390\.2390\.1920\.1920\.0900\.2040\.204Centroid Guess0\.2200\.2200\.1790\.1790\.1490\.1490\.0690\.1540\.1540\.2200\.2200\.1790\.1790\.1490\.1490\.0690\.1540\.154Sample\-level MI0\.0740\.0840\.0840\.0640\.0640\.0290\.0630\.0630\.2140\.2140\.2000\.2000\.0910\.0590\.1410\.141Our Method: Unsupervised SoftmaxReal Trajectory0\.2950\.2950\.3210\.3210\.1800\.1800\.0610\.2140\.2140\.2050\.2050\.1520\.1520\.1680\.1680\.0920\.1540\.154SLERP0\.2020\.2020\.1420\.1420\.1060\.1060\.0680\.1300\.1300\.3620\.3620\.3080\.3080\.2620\.2620\.1250\.2640\.264LERP0\.0910\.0910\.1180\.1180\.0860\.0860\.0490\.0860\.0860\.1370\.1370\.1850\.1850\.1310\.1310\.0640\.1300\.130TIES0\.0910\.0910\.1180\.1180\.0870\.0870\.0490\.0860\.0860\.1380\.1380\.1750\.1750\.1300\.1300\.0640\.1270\.127Our Method: Supervised 2\-Layer MLPReal Trajectory0\.1420\.1420\.0490\.0490\.0480\.0480\.0550\.0740\.0740\.1510\.1510\.1270\.1370\.1370\.0690\.1210\.121SLERP0\.2090\.2090\.1100\.1100\.0940\.0940\.0880\.1300\.1300\.2320\.2320\.2110\.2110\.1730\.1730\.0750\.1720\.172LERP0\.0920\.0920\.0310\.0220\.0420\.0420\.0470\.1060\.1150\.1290\.0640\.104TIES0\.0870\.0340\.0230\.0400\.0460\.1080\.1300\.1300\.1350\.1350\.0630\.109
#### Baselines\.

We compare WARP against*three*baselines:

- •Random Guess: A mixture sampled fromΔK−1\\Delta^\{K\-1\}, serving as an uninformed lower bound\.
- •Centroid Guess: A uniform mixtureπ^=1K​𝟏\\hat\{\\pi\}=\\frac\{1\}\{K\}\\mathbf\{1\}, acting as the best constant predictor without observingθref\\theta\_\{\\text\{ref\}\}\.
- •Sample\-level MI: We score each probing example by its loss reduction fromθbase\\theta\_\{\\text\{base\}\}toθref\\theta\_\{\\text\{ref\}\}, sweep the reduction threshold to*maximize detection accuracy*, and aggregate the resulting per\-domain membership rates into a mixture estimate\. We treat this as the closest sample\-based dataset recovery to our weight\-space approach\.

To isolate the effect of*simulation*, we further include an*oracle variant*of WARP that operates on real intermediate checkpoints saved during fine\-tuning, in place of pseudo\-checkpoints from merging\. For the merging operatorℳ\\mathcal\{M\}, we evaluate three choices:SLERP\(spherical interpolation\),LERP\(linear interpolation\), andTIES\[[32](https://arxiv.org/html/2607.01686#bib.bib16)\]\.

#### Main Results\.

Table[1](https://arxiv.org/html/2607.01686#S4.T1)reports WARP’s recovery accuracy, supporting three observations\.First, weight\-space geometry carries a strong signal for estimation\.WARP’s best supervised variant attains0\.0460\.046MAE on BERT and0\.1040\.104on GPT\-2\-Small, reducing the strongest baseline’s error by35%35\\%and30%30\\%respectively—well below random guessing and sample\-level membership inference\.Second, unsupervised readout is competitive\.Even without a learned projector—which would require collecting an additional synthetic training set—the parameter\-free softmax readout already outperforms all baselines and closes most of the gap to the supervised variant\.Third, simulated trajectories match or surpass real ones\.On BERT, supervised LERP and TIES both reach0\.0480\.048avg\. MAE versus0\.0800\.080for the oracle on real checkpoints; on GPT\-2\-Small, LERP reaches0\.1170\.117versus0\.1380\.138\. We attribute this to a*smoothness gap*: real fine\-tuning paths can be noisy—stochastic mini\-batches, learning\-rate warmup, and curriculum effects inject variance into per\-step alignment scores\. LERP and TIES instead create a clean monotone path fromθbase\\theta\_\{\\text\{base\}\}toθref\\theta\_\{\\text\{ref\}\}, yielding a smoother footprint matrixMMfrom which the 2\-layer MLP can more easily extract the mixture\. Overall, we demonstrate the feasibility of leveraging weight\-space geometry and the dataset footprints to reverse\-engineer the training mixture\.

Table 2:Domain mixture recovery error on GPT\-2\-Small \(using AG News and TIES merging\) across different training stages\. Both methods remain robust at recovering different types of checkpoint, with the supervised MLP consistently outperforming all baselines\. Lower is better\.
#### Different Training Recipes\.

An interesting question when deploying our method is which reference checkpoint to distill alignment matrices from, since the right choice is often unclear in practice—practitioners may stop training early to save compute, train to convergence, or inadvertently overtrain\. Table[2](https://arxiv.org/html/2607.01686#S4.T2)evaluates this sensitivity by extracting geometric footprints at three different checkpoints along the GPT\-2\-Small fine\-tuning trajectory on AG News: including short runs \(9 epochs, mirroring early stopping\), converged checkpoints \(12 epochs\), and overtrained checkpoints \(18 epochs, well past convergence\)\. Across all three stages, both our unsupervised softmax and supervised MLP variants outperform the baselines, with the supervised MLP achieving the lowest recovery error\. This demonstrates our framework’s robustness when facing diverse training recipes\.

## 5Conclusion

In this work, we introduce WARP, a framework that recovers a fine\-tuned model’s training domain mixture from its weights alone\. By using model merging to simulate the missing training trajectory, WARP extracts a geometric footprint of the training data and maps it to domain proportions\. In controlled experiments, WARP achieves MAE as low as 0\.046 on BERT and 0\.104 on GPT\-2, outperforming membership inference baselines and even an oracle with access to the true trajectory\. Overall, we demonstrate the feasibility of recovering a model’s training distribution from its weight\-space geometry—a step toward greater transparency in an era where model weights are shared but data recipes are not\.

## References

- \[1\]S\. K\. Ainsworth, J\. Hayase, and S\. Srinivasa\(2022\)Git re\-basin: merging models modulo permutation symmetries\.arXiv preprint arXiv:2209\.04836\.Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px3.p1.1)\.
- \[2\]A\. Albalak, L\. Pan, C\. Raffel, and W\. Y\. Wang\(2023\)Efficient online data mixing for language model pre\-training\.arXiv preprint arXiv:2312\.02406\.Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px1.p1.1)\.
- \[3\]G\. Cazenavette, T\. Wang, A\. Torralba, A\. A\. Efros, and J\. Zhu\(2022\)Dataset distillation by matching training trajectories\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 4750–4759\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p4.1)\.
- \[4\]M\. F\. Chen, M\. Y\. Hu, N\. Lourie, K\. Cho, and C\. Ré\(2024\)Aioli: a unified optimization framework for language model data mixing\.arXiv preprint arXiv:2411\.05735\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p1.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px1.p1.1)\.
- \[5\]M\. F\. Chen, T\. Murray, D\. Heineman, M\. Jordan, H\. Hajishirzi, C\. Ré, L\. Soldaini, and K\. Lo\(2026\)Olmix: a framework for data mixing throughout lm development\.arXiv preprint arXiv:2602\.12237\.Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]Z\. Chen, Y\. Miao, D\. Xiong,et al\.\(2026\)Data mixing for large language models pretraining: a survey and outlook\.arXiv preprint arXiv:2604\.16380\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p1.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p6.2)\.
- \[8\]M\. Duan, A\. Suri, N\. Mireshghallah, S\. Min, W\. Shi, L\. Zettlemoyer, Y\. Tsvetkov, Y\. Choi, D\. Evans, and H\. Hajishirzi\(2024\)Do membership inference attacks work on large language models?\.arXiv preprint arXiv:2402\.07841\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p3.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px2.p1.1)\.
- \[9\]S\. Fan, M\. Pagliardini, and M\. Jaggi\(2023\)Doge: domain reweighting with generalization estimation\.arXiv preprint arXiv:2310\.15393\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p1.1),[§1](https://arxiv.org/html/2607.01686#S1.p3.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]W\. Fu, H\. Wang, C\. Gao, G\. Liu, Y\. Li, and T\. Jiang\(2024\)Membership inference attacks against fine\-tuned large language models via self\-prompt calibration\.Advances in Neural Information Processing Systems37,pp\. 134981–135010\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p3.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px2.p1.1)\.
- \[11\]A\. Ge, T\. Huang, J\. Cooper, A\. Trost, Z\. Chu, S\. S\. S\. N\. GNVV, Z\. Cai, K\. Park, N\. Roberts, and F\. Sala\(2025\)R&B: domain regrouping and data mixture balancing for efficient foundation model training\.arXiv preprint arXiv:2505\.00358\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p1.1),[§1](https://arxiv.org/html/2607.01686#S1.p3.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]C\. Goddard, S\. Siriwardhana, M\. Ehghaghi, L\. Meyers, V\. Karpukhin, B\. Benedict, M\. McQuade, and J\. Solawetz\(2024\-11\)Arcee’s MergeKit: a toolkit for merging large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,F\. Dernoncourt, D\. Preoţiuc\-Pietro, and A\. Shimorina \(Eds\.\),Miami, Florida, US,pp\. 477–485\.External Links:[Link](https://aclanthology.org/2024.emnlp-industry.36),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-industry.36)Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2607.01686#S3.SS1.p1.5)\.
- \[13\]S\. Golchin and M\. Surdeanu\(2023\)Data contamination quiz: A tool to detect and estimate contamination in large language models\.CoRRabs/2311\.06233\.External Links:[Link](https://doi.org/10.48550/arXiv.2311.06233),[Document](https://dx.doi.org/10.48550/ARXIV.2311.06233),2311\.06233Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p2.1)\.
- \[14\]S\. Golchin and M\. Surdeanu\(2023\)Time travel in llms: tracing data contamination in large language models\.CoRRabs/2308\.08493\.External Links:[Link](https://doi.org/10.48550/arXiv.2308.08493),[Document](https://dx.doi.org/10.48550/ARXIV.2308.08493),2308\.08493Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p2.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px2.p1.1)\.
- \[15\]Y\. He, Y\. Hu, Y\. Lin, T\. Zhang, and H\. Zhao\(2024\)Localize\-and\-stitch: efficient model merging via sparse task arithmetic\.arXiv preprint arXiv:2408\.13656\.Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px3.p1.1)\.
- \[16\]T\. Huang, M\. Bilkhu, J\. Cooper, F\. Sala, and J\. Movellan\(2025\)Evaluating sample utility for efficient data selection by mimicking model weights\.arXiv preprint arXiv:2501\.06708\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p4.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2607.01686#S3.SS2.SSS0.Px1.p1.6)\.
- \[17\]G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi\(2022\)Editing models with task arithmetic\.arXiv preprint arXiv:2212\.04089\.Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px3.p1.1)\.
- \[18\]K\. Jordan, H\. Sedghi, O\. Saukh, R\. Entezari, and B\. Neyshabur\(2022\)Repair: renormalizing permuted activations for interpolation repair\.arXiv preprint arXiv:2211\.08403\.Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px3.p1.1)\.
- \[19\]F\. Kang, Y\. Sun, B\. Wen, S\. Chen, D\. Song, R\. Mahmood, and R\. Jia\(2024\)Autoscale: scale\-aware data mixing for pre\-training llms\.arXiv preprint arXiv:2407\.20177\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p3.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px1.p1.1)\.
- \[20\]Q\. Liu, X\. Zheng, N\. Muennighoff, G\. Zeng, L\. Dou, T\. Pang, J\. Jiang, and M\. Lin\(2025\)Regmix: data mixture as regression for language model pre\-training\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 38305–38339\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p3.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px1.p1.1)\.
- \[21\]I\. Magar and R\. Schwartz\(2022\)Data contamination: from memorization to exploitation\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 157–165\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p2.1)\.
- \[22\]P\. Maini, H\. Jia, N\. Papernot, and A\. Dziedzic\(2024\)LLM dataset inference: did you train on my dataset?\.Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px2.p1.1)\.
- \[23\]J\. Mattern, F\. Mireshghallah, Z\. Jin, B\. Schölkopf, M\. Sachan, and T\. Berg\-Kirkpatrick\(2023\)Membership inference attacks against language models via neighbourhood comparison\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 11330–11343\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p3.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px2.p1.1)\.
- \[24\]J\. X\. Morris, J\. O\. Yin, W\. Kim, V\. Shmatikov, and A\. M\. Rush\(2025\)Approximating language model training data from weights\.arXiv preprint arXiv:2506\.15553\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p4.1)\.
- \[25\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p6.2)\.
- \[26\]R\. Shokri, M\. Stronati, C\. Song, and V\. Shmatikov\(2017\)Membership inference attacks against machine learning models\.External Links:1610\.05820,[Link](https://arxiv.org/abs/1610.05820)Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p3.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px2.p1.1)\.
- \[27\]M\. Shukor, L\. Bethune, D\. Busbridge, D\. Grangier, E\. Fini, A\. El\-Nouby, and P\. Ablin\(2026\)Scaling laws for optimal data mixtures\.Advances in Neural Information Processing Systems38,pp\. 129554–129579\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p3.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px1.p1.1)\.
- \[28\]C\. Singh, J\. P\. Inala, M\. Galley, R\. Caruana, and J\. Gao\(2024\)Rethinking interpretability in the era of large language models\.arXiv preprint arXiv:2402\.01761\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p2.1)\.
- \[29\]B\. Wen, S\. Salekin, F\. Kang, B\. Howe, L\. L\. Wang, J\. Movellan, and M\. Bilkhu\(2026\)MixAtlas: uncertainty\-aware data mixture optimization for multimodal llm midtraining\.arXiv preprint arXiv:2604\.14198\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p3.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px1.p1.1)\.
- \[30\]M\. Wortsman, G\. Ilharco, S\. Y\. Gadre, R\. Roelofs, R\. Gontijo\-Lopes, A\. S\. Morcos, H\. Namkoong, A\. Farhadi, Y\. Carmon, S\. Kornblith,et al\.\(2022\)Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.InInternational conference on machine learning,pp\. 23965–23998\.Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px3.p1.1)\.
- \[31\]S\. M\. Xie, H\. Pham, X\. Dong, N\. Du, H\. Liu, Y\. Lu, P\. S\. Liang, Q\. V\. Le, T\. Ma, and A\. W\. Yu\(2023\)Doremi: optimizing data mixtures speeds up language model pretraining\.Advances in Neural Information Processing Systems36,pp\. 69798–69818\.Cited by:[§1](https://arxiv.org/html/2607.01686#S1.p1.1),[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px1.p1.1)\.
- \[32\]P\. Yadav, D\. Tam, L\. Choshen, C\. A\. Raffel, and M\. Bansal\(2023\)Ties\-merging: resolving interference when merging models\.Advances in neural information processing systems36,pp\. 7093–7115\.Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2607.01686#S3.SS1.p1.5),[§4](https://arxiv.org/html/2607.01686#S4.SS0.SSS0.Px2.p2.1)\.
- \[33\]E\. Yang, L\. Shen, G\. Guo, X\. Wang, X\. Cao, J\. Zhang, and D\. Tao\(2026\-02\)Model merging in llms, mllms, and beyond: methods, theories, applications, and opportunities\.ACM Comput\. Surv\.58\(8\)\.External Links:ISSN 0360\-0300,[Link](https://doi.org/10.1145/3787849),[Document](https://dx.doi.org/10.1145/3787849)Cited by:[§3\.1](https://arxiv.org/html/2607.01686#S3.SS1.p1.5)\.
- \[34\]E\. Yang, L\. Shen, G\. Guo, X\. Wang, X\. Cao, J\. Zhang, and D\. Tao\(2026\)Model merging in llms, mllms, and beyond: methods, theories, applications, and opportunities\.ACM Computing Surveys58\(8\),pp\. 1–41\.Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px3.p1.1)\.
- \[35\]P\. Young, A\. Lai, M\. Hodosh, and J\. Hockenmaier\(2014\)From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions\.Transactions of the Association for Computational Linguistics2,pp\. 67–78\.External Links:[Link](https://aclanthology.org/Q14-1006),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00166)Cited by:[§4](https://arxiv.org/html/2607.01686#S4.SS0.SSS0.Px1.p1.16)\.
- \[36\]L\. Yu, B\. Yu, H\. Yu, F\. Huang, and Y\. Li\(2024\)Language models are super mario: absorbing abilities from homologous models as a free lunch\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2607.01686#S2.SS0.SSS0.Px3.p1.1)\.
- \[37\]X\. Zhang, J\. Zhao, and Y\. LeCun\(2015\)Character\-level convolutional networks for text classification\.Advances in neural information processing systems28\.Cited by:[§4](https://arxiv.org/html/2607.01686#S4.SS0.SSS0.Px1.p1.16)\.

## Appendix Roadmap

Our appendix is structured as follows\. We use Appendix[A](https://arxiv.org/html/2607.01686#A1)to summarize our full framework in pseudocode\. Appendix[B](https://arxiv.org/html/2607.01686#A2)documents the experimental setup and training configurations\. Finally, Appendix[C](https://arxiv.org/html/2607.01686#A3)outlines directions for future work\.

## Appendix AAlgorithm

Next, we summarize the full WARP procedure in Algorithm[1](https://arxiv.org/html/2607.01686#alg1)\. WARP proceeds in three stages: simulating a trajectory betweenθbase\\theta\_\{\\text\{base\}\}andθref\\theta\_\{\\text\{ref\}\}via model merging \(Sec\.[3\.1](https://arxiv.org/html/2607.01686#S3.SS1)\), distilling each pseudo\-checkpoint into a domain\-level geometric footprint via Mimic Scores \(Sec\.[3\.2](https://arxiv.org/html/2607.01686#S3.SS2)\), and mapping the resulting footprint matrix to a predicted mixture through either an unsupervised softmax readout or a supervised projector trained on synthetic pairs \(Sec\.[3\.3](https://arxiv.org/html/2607.01686#S3.SS3)\)\.

Algorithm 1WARP: Weight\-Space Analysis for Recovering Data Portfolios1:Base model

𝜽base\\bm\{\\theta\}\_\{\\text\{base\}\}, fine\-tuned model

𝜽ref\\bm\{\\theta\}\_\{\\text\{ref\}\}, data source

SSover

KKdomains, merging operator

ℳ\\mathcal\{M\}with schedule

\{αt\}t=1T\\\{\\alpha\_\{t\}\\\}\_\{t=1\}^\{T\}, mode

∈\{Unsup,Sup\}\\in\\\{\\textsc\{Unsup\},\\textsc\{Sup\}\\\}, temperature

τ\\tau, \(Suponly\) mixture distribution

Π\\Piand \# pairs

JJ\.

2:Estimated domain mixture

𝝅^∈ΔK−1\\hat\{\\bm\{\\pi\}\}\\in\\Delta^\{K\-1\}\.

3:Initialize:probing dataset

Dprobe=⋃k=1KDkD\_\{\\text\{probe\}\}=\\bigcup\_\{k=1\}^\{K\}D\_\{k\}sampled from

SSwith known labels

4:

5:// Stage 1: Simulate trajectory // Stage 2: Distill footprint

6:for

t=1t=1to

TTdo

7:

𝜽~t←ℳ​\(𝜽base,𝜽ref;αt\)\\tilde\{\\bm\{\\theta\}\}\_\{t\}\\leftarrow\\mathcal\{M\}\(\\bm\{\\theta\}\_\{\\text\{base\}\},\\,\\bm\{\\theta\}\_\{\\text\{ref\}\};\\,\\alpha\_\{t\}\),

𝐯t←𝜽ref−𝜽~t\\mathbf\{v\}\_\{t\}\\leftarrow\\bm\{\\theta\}\_\{\\text\{ref\}\}\-\\tilde\{\\bm\{\\theta\}\}\_\{t\}⊳\\trianglerightPseudo\-checkpoint and direction

8:

mi,t←⟨−∇ℓ​\(xi,yi;𝜽~t\),𝐯t⟩/‖𝐯t‖m\_\{i,t\}\\leftarrow\\langle\-\\nabla\\ell\(x\_\{i\},y\_\{i\};\\,\\tilde\{\\bm\{\\theta\}\}\_\{t\}\),\\,\\mathbf\{v\}\_\{t\}\\rangle\\,/\\,\\\|\\mathbf\{v\}\_\{t\}\\\|for

\(xi,yi\)∈Dprobe\(x\_\{i\},y\_\{i\}\)\\in D\_\{\\text\{probe\}\}⊳\\trianglerightMimic Score

9:Min\-max normalize

\{mi,t\}i\\\{m\_\{i,t\}\\\}\_\{i\};

m¯k,t←1\|Dk\|​∑i∈Dkmi,t\\bar\{m\}\_\{k,t\}\\leftarrow\\frac\{1\}\{\|D\_\{k\}\|\}\\sum\_\{i\\in D\_\{k\}\}m\_\{i,t\}⊳\\trianglerightPool by domain

10:endfor

11:

M←\[m¯k,t\]K×TM\\leftarrow\[\\bar\{m\}\_\{k,t\}\]\_\{K\\times T\}⊳\\trianglerightGeometric footprint

12:

13:// Stage 3: Map footprint to mixture

14:ifMode isUnsupthen

15:

π^k←softmaxk​\(1T​∑tm¯k,t/τ\)\\hat\{\\pi\}\_\{k\}\\leftarrow\\mathrm\{softmax\}\_\{k\}\\\!\\left\(\\tfrac\{1\}\{T\}\\sum\_\{t\}\\bar\{m\}\_\{k,t\}\\,/\\,\\tau\\right\)⊳\\trianglerightLabel\-free readout

16:else

17:for

j=1j=1to

JJdo

18:Sample

𝝅\(j\)∼Π\\bm\{\\pi\}^\{\(j\)\}\\sim\\Pi, draw

D\(j\)D^\{\(j\)\}from

SSwith mixture

𝝅\(j\)\\bm\{\\pi\}^\{\(j\)\}, fine\-tune

𝜽base\\bm\{\\theta\}\_\{\\text\{base\}\}on

D\(j\)D^\{\(j\)\}to get

𝜽ref\(j\)\\bm\{\\theta\}^\{\(j\)\}\_\{\\text\{ref\}\}
19:

M\(j\)←M^\{\(j\)\}\\leftarrowStages 1–2 with

\(𝜽base,𝜽ref\(j\),Dprobe\)\(\\bm\{\\theta\}\_\{\\text\{base\}\},\\,\\bm\{\\theta\}^\{\(j\)\}\_\{\\text\{ref\}\},\\,D\_\{\\text\{probe\}\}\)
20:endfor

21:Train

fϕ:ℝK×T→ΔK−1f\_\{\\phi\}:\\mathbb\{R\}^\{K\\times T\}\\to\\Delta^\{K\-1\}on

\{\(M\(j\),𝝅\(j\)\)\}j=1J\\\{\(M^\{\(j\)\},\\,\\bm\{\\pi\}^\{\(j\)\}\)\\\}\_\{j=1\}^\{J\};

𝝅^←fϕ​\(M\)\\hat\{\\bm\{\\pi\}\}\\leftarrow f\_\{\\phi\}\(M\)
22:endif

23:return

𝝅^\\hat\{\\bm\{\\pi\}\}

## Appendix BExperimental Details

Here we provide additional details of our experimental setup\. We discuss \(i\) how reference models are constructed, \(ii\) how the probing dataset is sampled, \(iii\) the supervised projector training pipeline, and \(iv\) the computational resources used\.

#### Constructing Reference Models\.

We validate WARP in a controlled setting where the ground\-truth mixtureπ⋆\\pi^\{\\star\}is known\. For each dataset, we sample4040mixtures\{πi⋆\}i=140\\\{\\pi^\{\\star\}\_\{i\}\\\}\_\{i=1\}^\{40\}from the simplexΔK−1\\Delta^\{K\-1\}\. For eachπi⋆\\pi^\{\\star\}\_\{i\}, we draw5,0005\{,\}000training examples whose per\-domain proportions followπi⋆\\pi^\{\\star\}\_\{i\}, and fine\-tune bothBERTandGPT\-2\-Smallfor99epochs starting from their publicly released pretrained weights, which we use asθbase\\theta\_\{\\text\{base\}\}\. For each reference model, we sweep the learning rate over\{6​e−6,1​e−5,3​e−5,5​e−5,1​e−4\}\\\{6\\mathrm\{e\}\{\-\}6,\\ 1\\mathrm\{e\}\{\-\}5,\\ 3\\mathrm\{e\}\{\-\}5,\\ 5\\mathrm\{e\}\{\-\}5,\\ 1\\mathrm\{e\}\{\-\}4\\\}and select the value that minimizes training loss to obtainθref\\theta\_\{\\text\{ref\}\}\. This yields4040reference models per \(dataset, architecture\) pair\.

#### Creating Probing Dataset\.

WARP makes a simple assumption about the domain distribution of the probing set; it requires only thatDprobeD\_\{\\text\{probe\}\}shares the sameKKdomains asDrefD\_\{\\text\{ref\}\}\. We sample\|Dprobe\|=2,500\|D\_\{\\text\{probe\}\}\|=2\{,\}500examples fromSS, with2,500/K2\{,\}500/Kexamples drawn uniformly per domain\. The probing dataset is fixed once per \(dataset, architecture\) pair and reused across all4040mixtures\. For each probing example, we compute its alignment score from the per\-sample gradient and the directional vector pointing from each pseudo\-checkpoint towardθref\\theta\_\{\\text\{ref\}\}\.

#### Training Projector\.

For the supervised variant, the projector is a22\-layer MLP with ReLU activation, and a softmax output layer that produces a vector onΔK−1\\Delta^\{K\-1\}\. The input is the matrix of domain\-level alignment statistics aggregated fromDprobeD\_\{\\text\{probe\}\}, and the target is the ground\-truth mixtureπ⋆\\pi^\{\\star\}\. We train it with a learning rate1​e−41\\mathrm\{e\}\{\-\}4for200200epochs\. Since we have4040\(mixture, reference\-model\) pairs per \(dataset, architecture\), we report results under55\-fold cross\-validation: in each fold,3232pairs train the projector and the held\-out88pairs are used for evaluation\. All the experiments are conducted on an NVIDIA RTX A6000 GPU\.

#### Evaluation\.

We report MAE betweenπ^\\hat\{\\pi\}andπ⋆\\pi^\{\\star\}, averaged over the4040ground\-truth mixtures in Table[1](https://arxiv.org/html/2607.01686#S4.T1)\. We additionally report mean squared error \(MSE\) in Table[3](https://arxiv.org/html/2607.01686#A2.T3)\.

Table 3:Domain mixture recovery error \(MSE between predictedπ^\\hat\{\\pi\}and ground\-truthπ⋆\\pi^\{\\star\}, averaged over4040trials\) on BERT and GPT\-2\-Small across datasets\. Lower is better\. Best results within each model block arebolded; second\-best areunderlined\.BERTGPT\-2\-SmallMethodSNLIAGNewsYelpYahooAvg\.SNLIAGNewsYelpYahooAvg\.\(3 cls\.\)\(4 cls\.\)\(5 cls\.\)\(10 cls\.\)MSE\(3 cls\.\)\(4 cls\.\)\(5 cls\.\)\(10 cls\.\)MSEBaselinesRandom Guess0\.1400\.1400\.0950\.0950\.0660\.0660\.0140\.0790\.0790\.1400\.1400\.0950\.0950\.0660\.0660\.0140\.0140\.0790\.079Centroid Guess0\.0720\.0720\.0480\.0480\.0320\.0320\.0070\.0400\.0400\.0720\.0720\.0480\.0480\.0320\.0070\.0400\.040Sample\-level MI0\.0090\.0120\.0120\.0070\.0070\.0010\.0070\.0070\.0730\.0730\.0680\.0680\.0170\.0080\.0420\.042Our Method: Unsupervised SoftmaxReal Trajectory0\.1560\.1560\.1630\.1630\.0620\.0620\.0070\.0970\.0970\.0700\.0700\.0460\.0460\.0580\.0580\.0190\.0190\.0480\.048SLERP0\.0710\.0710\.0290\.0290\.0190\.0190\.0080\.0320\.0320\.1970\.1970\.1630\.1630\.1210\.1210\.0360\.0360\.1300\.130LERP0\.0130\.0210\.0210\.0130\.0130\.0040\.0130\.0130\.0360\.0360\.0620\.0620\.0320\.0080\.0350\.035TIES0\.0130\.0210\.0210\.0130\.0130\.0040\.0130\.0130\.0360\.0360\.0530\.0530\.0320\.0080\.0320\.032Our Method: Supervised 2\-Layer MLPReal Trajectory0\.0400\.0400\.0050\.0050\.0060\.0140\.0140\.0400\.0400\.0300\.0400\.0400\.0090\.0090\.0300\.030SLERP0\.0700\.0700\.0180\.0180\.0180\.0180\.0140\.0340\.0340\.0800\.0800\.0770\.0770\.0530\.0530\.0110\.0110\.0550\.055LERP0\.0170\.0170\.0020\.0010\.0040\.0060\.0220\.0240\.0360\.0360\.0080\.023TIES0\.0140\.0140\.0020\.0010\.0030\.0050\.0230\.0330\.0330\.0370\.0370\.0080\.025

## Appendix CFuture Work

WARP provides the*first*mechanism for recovering the domain mixture of a fine\-tuned model from its released weights alone\. We outline several directions that can extend its scope, including \(i\) interpreting the learned projector, \(ii\) strengthening the unsupervised readout, and \(iii\) scaling to large\-scale LLMs\.

#### Interpretability of the Learned Projector\.

The supervised variant of WARP encodes, in the weights of its MLP projector, an*implicit*model of how Mimic Scores across pseudo\-checkpoints map to domain proportions\. Understanding this mapping would clarify*which*stages of the simulated trajectory carry the strongest signal for each domain, and*why*\. A starting point is to replace the MLP with shallower, fully linear projectors, whose coefficients can be read directly as per\-stage importance weights\. Combined with a larger set of synthetic pseudo\-checkpoints and explicit regularization \(e\.g\., sparsity or smoothness acrosstt\), this could begin to expose the temporal structure of domain influence as the model evolves fromθbase\\theta\_\{\\text\{base\}\}towardθref\\theta\_\{\\text\{ref\}\}\.

#### Advanced Unsupervised Readouts\.

The current unsupervised variant uses two assumptions: that uniform averaging of pooled Mimic Scores across pseudo\-checkpoints is a*reliable*aggregation, and that the resulting per\-domain quantities behave like logits suitable for a softmax\. A more principled alternative would treat the readout as inference in a graphical model over pseudo\-checkpoints, where each step contributes a noisy vote on the underlying mixture and*weak supervision*techniques infer per\-stage reliabilities without labels\. Such unsupervised techniques could adaptively down\-weight uninformative stages, potentially narrowing the gap to the supervised variant while preserving the label\-free setting\.

#### Scaling to Large\-Scale LLMs\.

Finally, our controlled experiments fine\-tune BERT and GPT\-2 under known mixtures, which isolates the recovery problem but leaves open how WARP behaves at the scale of modern LLM training\. LLMs are typically adapted through long, multi\-stage pipelines—continued mid\-training, supervised fine\-tuning, and preference optimization—over corpora spanning dozens of domains\. Extending WARP to these regimes raises two questions in particular: whether linear interpolation remains a useful proxy for trajectories that traverse qualitatively different objectives, and how the geometric footprint behaves when domains are numerous and more complex\. These directions will help extend WARP from a controlled diagnostic tool toward a recovery tool applicable to frontier models\.