Spectral Signatures of Large Language Models

# Spectral Signatures of Large Language Models
Source: [https://arxiv.org/html/2607.03377](https://arxiv.org/html/2607.03377)
\(2026\)

###### Abstract\.

The rapidly growing repository of publicly available large language models \(LLMs\) presents significant challenges for systematic management and quantification at scale, such as model lineage tracing, licensing, and evaluation\. However, task\-specific benchmarks are insufficient for this setting, as LLMs differ widely in architectures, scales, and training procedures\. To address this challenge, we adopt*spectral shape*\-based metrics for managing and quantifying LLMs based on Heavy\-Tailed Self\-Regularization theory\. Our approach uses the shape information of the weight empirical spectral density as a compact spectral signature of each model\. This signature captures intrinsic properties of pretrained models and remains robust during post\-training, making it suitable for model\-level analysis\. In addition, this metric is*data\-free*,*computationally\-efficient*, and*scale\-invariant*, enabling large\-scale analysis in practice\. Moreover, we curate a large and diverse model corpus consisting of major open\-source LLM families, and use it to systematically benchmark spectral and non\-spectral metrics across models and downstream tasks\. We show that our spectral signature supports the tracking of the model lineage, the unsupervised clustering of similar models, and the quantification of the model performance\. Overall, the proposed spectral signature provides a meaningful proxy for broad performance trends across LLMs, enabling efficient organization, comparison, and analysis of large model collections\. The code is available at[https://github\.com/Ingrid\-505/Spectral\_Signature](https://github.com/Ingrid-505/Spectral_Signature)\.

Large language models; Model lineage; Model ranking; Weight space learning

††journalyear:2026††copyright:cc††conference:Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2; August 09–13, 2026; Jeju Island, Republic of Korea††booktitle:Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2 \(KDD ’26\), August 09–13, 2026, Jeju Island, Republic of Korea††doi:10\.1145/3770855\.3818090††isbn:979\-8\-4007\-2259\-2/2026/08## 1\.Introduction

Publicly available large language models \(LLMs\) have been growing rapidly in recent years, with hundreds of thousands of models now available through open\-source platforms such as Hugging Face \(HF\)\(Zhuanget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib28)\)\. These models exhibit significant diversity in their training trajectories, including variations in model architecture, parameter scale, training data, and empirical performance\(Horwitzet al\.,[2025a](https://arxiv.org/html/2607.03377#bib.bib4)\)\. As the ecosystem grows, there is an increasing need for reliable and efficient methods to characterize, compare, and manage LLMs beyond task\-level benchmarks\. Representative examples include identifying whether deployed LLM systems can be traced to the same backbone foundation model for intellectual property and safety purposes\(Zhuet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib29)\), as well as tracking and organizing model variants and versions in large\-scale LLM databases\(Horwitzet al\.,[2025b](https://arxiv.org/html/2607.03377#bib.bib5)\)\. Moreover, LLMs present distinct idiosyncrasies across downstream tasks, with different models specializing in different areas such as code generation, mathematical reasoning, long\-context processing, or interactive dialogue\. These considerations point to the need for a*principled*and*data\-light*framework to trace model sources, capture model idiosyncrasies, and quantify model performances for large\-scale LLM collections\.

In this work, we introduce a*spectral shape*\-based perspective for LLM quantification that builds on insights from Heavy\-Tailed Self\-Regularization \(HT\-SR\) theory\(Martin and Mahoney,[2021](https://arxiv.org/html/2607.03377#bib.bib13); Martinet al\.,[2021](https://arxiv.org/html/2607.03377#bib.bib36); Martin and Hinrichs,[2025](https://arxiv.org/html/2607.03377#bib.bib37)\)\. HT\-SR theory analyzes the eigenspectrum of weight matrices, and prior studies\(Yanget al\.,[2023](https://arxiv.org/html/2607.03377#bib.bib24); Martinet al\.,[2021](https://arxiv.org/html/2607.03377#bib.bib36)\)have shown that the shape of the empirical spectral density \(ESD\) encodes informative signals about model capacity, quality, and training dynamics that are not well captured by scale\-based measures\. Moreover, ESD shape metrics are data\-independent, computationally lightweight, and remain robust under noise and scaling\. These properties make them particularly well\-suited to large\-scale LLM settings, where task\-specific evaluation can be expensive\. An overview of different quantification metrics is shown in Tab\.[1](https://arxiv.org/html/2607.03377#S1.T1)\.

We use the shape metric of the ESD, namely PL\_Alpha\_Hill, as a*spectral signature*of each LLM\. The PL\_Alpha\_Hill metric summarizes the global structure of layer\-wise weight eigenspectra into a compact representation, enabling efficient storage and comparison across large model collections\. We adopt this spectral signature to support three downstream tasks: supervised model classification, unsupervised clustering of LLMs, and quantifying LLM performance\. Our contributions are summarized below\.

- •We introduce a shape\-based metric for reliably and efficiently characterizing, comparing, and quantifying LLMs beyond task\-level benchmarks\. Our metric is*scale\-invariant*,*data\-free*,*theory\-driven*, and*easy\-to\-compute*\.
- •We curate a diverse and representative LLM corpus of up to 499 models and use it to benchmark three tasks: LLM similarity measurement, model classification and clustering, and performance prediction\. We further conduct a systematic comparison of our spectral signature against a range of baselines across various experimental settings, demonstrating*consistent effectiveness*across tasks and conditions\.
- •We compare shape\- and scale\-based metrics under output\-invariant reparameterizations and noise perturbations\. ESD shape metrics remain stable, while scale\-based baselines degrade, indicating that spectral shape captures intrinsic weight structure for reliable LLM quantification\.

Table 1\.Comparisons of different metrics for LLM tasks\.✓and✗denote strong and weak performance, respectively\.MetricsScale\-invariantData\-freeEasy to computeScalableTheory\-drivenClassificationClusteringQuantificationLogits\(Yang and Wu,[2024](https://arxiv.org/html/2607.03377#bib.bib53)\)✓✗✓✗✗✓✓✗REEF\(Zhanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib48)\)✓✗✓✗✗✓✓✗PCS\(Zenget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib3)\)✗✓✓✗✗✓✓✗Spectral norm✗✓✓✓✓✓✓✓Log alpha norm✗✓✓✓✓✓✓✓Shape metric \(Ours\)✓✓✓✓✓✓✓✓

## 2\.Related Work

#### Model Lineage and Independence Testing\.

The growing population of open\-source LLMs has spurred research into determining LLM lineage—whether two models are independently trained or one is trained from another—which is crucial for intellectual property and model safety\. Recent works use distinct approaches: output\-based methods use output comparison\(Nikolicet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib30)\), fingerprint queries\(Jinet al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib27)\), and lexical cues\(Sunet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib2); Wuet al\.,[2023](https://arxiv.org/html/2607.03377#bib.bib31)\)to compare models without requiring parameter access; watermarking techniques inject provenance information during training or decoding\(Wang and Kerschbaum,[2019](https://arxiv.org/html/2607.03377#bib.bib34); Uchidaet al\.,[2017](https://arxiv.org/html/2607.03377#bib.bib35); Guet al\.,[2022](https://arxiv.org/html/2607.03377#bib.bib33); Jianget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib32)\); whereas white\-box methods perform hypothesis testing\(Zhuet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib29)\)or fingerprinting using metrics derived from model weight structures\(Zenget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib3),[2025](https://arxiv.org/html/2607.03377#bib.bib42); Wanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib41); Yoonet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib39)\)\.

#### Model Zoos and LLM Embeddings\.

There has been growing interest in exploring representation\-based approaches to capture model dependence, properties, and relationships, enabling efficient organization and management of large model zoos\. Relevant directions include LLM embeddings\(Zhuanget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib28)\), weight\-space learning\(Horwitzet al\.,[2025a](https://arxiv.org/html/2607.03377#bib.bib4),[b](https://arxiv.org/html/2607.03377#bib.bib5); Guetaet al\.,[2023](https://arxiv.org/html/2607.03377#bib.bib6)\), and loss\-landscape\-based grouping\(Guetaet al\.,[2023](https://arxiv.org/html/2607.03377#bib.bib6); Schürholtet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib7)\)\. LLM embedding methods\(Zhuanget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib28)\)learn compact and model\-level representations that can be used to predict downstream performance across tasks without additional inference\. In contrast, weight\-space learning approaches operate directly on model parameters\. For example, Horwitz*et al\.*\(Horwitzet al\.,[2025a](https://arxiv.org/html/2607.03377#bib.bib4)\)combine weight\-space distances with model metadata to construct an “atlas” of open models, enabling similarity search and provenance analysis within large repositories\. A complementary line of work\(Schürholtet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib7)\)considers loss\-landscape and phase\-based information\(Yanget al\.,[2021](https://arxiv.org/html/2607.03377#bib.bib8)\), such as mode connectivity and CKA similarity, to partition models into distinct regimes and assess their similarity\.

#### HT\-SR Theory\.

HT\-SR theory is a theoretical framework based on random matrix theory for analyzing weight matrices geometry of neural networks \(NNs\)\(Mahoney and Martin,[2019](https://arxiv.org/html/2607.03377#bib.bib12); Martin and Mahoney,[2021](https://arxiv.org/html/2607.03377#bib.bib13); Couillet and Liao,[2022](https://arxiv.org/html/2607.03377#bib.bib14); Liao and Mahoney,[2025](https://arxiv.org/html/2607.03377#bib.bib15)\)\. HT\-SR theory was proposed based on the observation that well\-trained, state\-of\-the\-art NNs often exhibit HT structures in the ESD of each layer\. In the meantime, several rigorous theories in stochastic gradient descent relating HT phenomena to generalization performance were established, providing further theoretical support for HT\-SR theory\(Gurbuzbalabanet al\.,[2021](https://arxiv.org/html/2607.03377#bib.bib19); Hodgkinson and Mahoney,[2021](https://arxiv.org/html/2607.03377#bib.bib20); Hodgkinsonet al\.,[2022](https://arxiv.org/html/2607.03377#bib.bib21); Simsekliet al\.,[2019](https://arxiv.org/html/2607.03377#bib.bib22),[2020](https://arxiv.org/html/2607.03377#bib.bib23)\)\. Recent works extend theoretical foundations and show that HT\-SR Theory can be used to assess the training quality of modern deep NN models across various domains \(e\.g\., computer vision and natural language processing\) without accessing any training or test data\(Yanget al\.,[2023](https://arxiv.org/html/2607.03377#bib.bib24); Martin and Mahoney,[2021](https://arxiv.org/html/2607.03377#bib.bib13)\)\. Based on this insight, recent studies have further proposed HT\-SR theory\-based methods for practical NN applications, including hyperparameter tuning\(Zhouet al\.,[2023](https://arxiv.org/html/2607.03377#bib.bib1); Liuet al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib16); Huet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib17)\)and model pruning\(Luet al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib18); Huet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib17)\)\. In this work, we use the HT\-SR theory to analyze the geometric similarities of weight matrices of LLMs\.

## 3\.Preliminaries

#### Problem Setup\.

In large LLM repositories, many checkpoints share a common pretrained backbone but differ through post\-training, distillation, pruning/quantization, or model merging\. Our goal is to perform*data\-free*lineage analysis: comparing models and tracing backbone relationships directly from their weights\. This requires robustness to output\-invariant transformations, such as uniform scaling, hidden\-unit permutations, and linear\-chain reparameterizations, which can make naive weight\-space similarities unreliable\. We therefore focus on spectral shape statistics of weight matrices, which capture intrinsic weight structures while being less sensitive to transformations that change the parameterization but preserve the model’s input\-output function\.

#### Weight Analysis with HT\-SR Theory\.

HT\-SR theory\(Martin and Mahoney,[2021](https://arxiv.org/html/2607.03377#bib.bib13)\)demonstrates the empirical fact that well\-trained NNs tend to exhibit strong correlations in weight matrices, resulting in a heavy\-tailed ESD of each weight matrix\. ESD characterizes the eigenvalue distribution of the correlation matrix𝐖⊤​𝐖\\mathbf\{W\}^\{\\top\}\\mathbf\{W\}of the weight matrix𝐖\\mathbf\{W\}\. HT\-SR theory can be used to analyze the dynamics of weight matrix ESDs during training, by modeling the tail component of the ESD with a power\-law \(PL\) distribution within the interval\(λm​i​n,λm​a​x\)\(\\lambda\_\{min\},\\lambda\_\{max\}\), and its tail indexα\\alpha:p​\(λ\)∝λ−α,λm​i​n<λ<λm​a​xp\(\\lambda\)\\propto\\lambda^\{\-\\alpha\},\\quad\\lambda\_\{min\}<\\lambda<\\lambda\_\{max\}\.

One commonly used method to approximate the tail indexα\\alphais with the Hill Estimator111It is known that more mathematically principled estimators exist, such as those based on MLE\(Alstottet al\.,[2014](https://arxiv.org/html/2607.03377#bib.bib62); Clausetet al\.,[2009](https://arxiv.org/html/2607.03377#bib.bib63)\)or eigenvalue repulsion\(Hodgkinsonet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib64)\)\. However, in our experiments, the Hill estimator demonstrates more stable performance and is easier to scale to large settings\.\(Hill,[1975](https://arxiv.org/html/2607.03377#bib.bib25)\), which computes an estimate PL\_Alpha\_Hill:

\(1\)PL\_Alpha\_Hill=1\+k∑i=1kln⁡λn−i\+1λn−k,\\text\{PL\\\_\\allowbreak Alpha\\\_\\allowbreak Hill\}=1\+\\frac\{k\}\{\\sum\_\{i=1\}^\{k\}\\operatorname\{ln\}\\frac\{\\lambda\_\{n\-i\+1\}\}\{\\lambda\_\{n\-k\}\}\},where\{λi\}i=1n\\\{\\lambda\_\{i\}\\\}\_\{i=1\}^\{n\}is sorted in ascending order\. For the selection ofkk, we adopt the approach from previous works\(Zhouet al\.,[2023](https://arxiv.org/html/2607.03377#bib.bib1); Yanget al\.,[2023](https://arxiv.org/html/2607.03377#bib.bib24)\), where we setk=n2k=\\frac\{n\}\{2\}\. PL\_Alpha\_Hill measures the shape of the eigenspectrum of each layer individually\. Unlike norm\-based metrics such as the spectral norm or RMS norm, PL\_Alpha\_Hill is robust under scaling\. Moreover, since permutations are orthogonal changes of basis, the spectrum of𝐖⊤​𝐖\\mathbf\{W\}^\{\\top\}\\mathbf\{W\}\(and hence its ESD shape\) is invariant to hidden\-unit permutations, motivating PL\_Alpha\_Hill as a stable building block for our spectral signatures \(Sec\.[4\.1](https://arxiv.org/html/2607.03377#S4.SS1)\)\. PL\_Alpha\_Hill also measures the dynamics of the ESD shape, and previous works\(Yanget al\.,[2023](https://arxiv.org/html/2607.03377#bib.bib24); Liuet al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib16)\)have shown that PL\_Alpha\_Hill reflects the training quality of different layers\. Layers with larger PL\_Alpha\_Hill tend to be “over\-trained,” while layers with smaller PL\_Alpha\_Hill tend to be relatively “under\-trained\.”

#### Robustness of ESD Shape Metrics in Post\-training\.

A critical prerequisite for a spectral signature to serve as a reliable lineage identifier is its robustness across downstream adaptations\. We validate the robustness of the layer\-wise PL\_Alpha\_Hill as a spectral signature by examining its stability under post\-training transformations\. We observe that the impact of post\-training on the weight ESD is minimal; the spectral signature is predominantly shaped during the pre\-training phase\. As shown in Fig\.[1](https://arxiv.org/html/2607.03377#S3.F1), the spectral signature exhibits distinct behaviors for intra\-family and inter\-family comparisons\. We observe a near\-perfect spectral overlap between Llama\-3\.1 and its post\-trained Tulu\-3 variant\. Simultaneously, the separation from the Llama\-2 baseline is clear\. This distinction supports that the spectral signature is primarily determined during pre\-training and remains robust to post\-training adjustments\. We provide additional results in Appendix[7\.1](https://arxiv.org/html/2607.03377#S7.SS1)to further corroborate these findings\.

![Refer to caption](https://arxiv.org/html/2607.03377v1/x1.png)Figure 1\.Eigenspectrum shape serves as a distinct signature for model lineage\. We visualize the weight ESDs of three LLMs\. Models sharing the same backbone \(Llama\-3\.1 and its post\-trained variant Tulu\-3\) exhibit nearly identical spectral shapes and power\-law exponents \(P​L​\_​A​l​p​h​a​\_​H​i​l​l≈2\.85PL\\\_\\allowbreak Alpha\\\_\\allowbreak Hill\\approx 2\.85\), while a model from a different family \(Llama\-2\) shows significant deviation\. This structural persistence validates the use of ESD shape as a robust spectral signature\.

## 4\.Method

Using robustness properties of PL\_Alpha\_Hill established in Sec\.[3](https://arxiv.org/html/2607.03377#S3), we present our methodology for applying PL\_Alpha\_Hill\-based spectral signatures to model quantification\. As illustrated in Fig\.[2](https://arxiv.org/html/2607.03377#S4.F2), for a given collection of LLMs, we extract the PL\_Alpha\_Hill statistic from each weight matrix and aggregate these measurements to construct a model\-level spectral signature, as formalized in Sec\.[4\.1](https://arxiv.org/html/2607.03377#S4.SS1)\. We then use these spectral signatures to address three model quantification tasks: \(1\) measuring similarity between LLMs, \(2\) model classification and clustering, and \(3\) predicting downstream LLM performance\. The detailed methodology is shown in Sec\.[4\.2](https://arxiv.org/html/2607.03377#S4.SS2)\.

![Refer to caption](https://arxiv.org/html/2607.03377v1/x2.png)Figure 2\.Overview of our proposed shape\-based spectral signature framework\. We analyze the eigenspectrum of model weights and use the shape information of ESDs \(PL\_Alpha\_Hill\) as spectral signatures for downstream tasks\.### 4\.1\.Formulation of Spectral Signatures

For a modelfθf\_\{\\theta\}in parameter spaceΘ\\ThetawithLLtransformer blocks andNmodN\_\{\\text\{mod\}\}modules, we quantify the model by constructing a spectral signature:𝐙∈ℝNmod×L\\mathbf\{Z\}\\in\\mathbb\{R\}^\{N\_\{\\text\{mod\}\}\\times L\}, which is a tensor consisting of the PL\_Alpha\_Hill measure of each module\. Each row𝐙i,:∈ℝL\\mathbf\{Z\}\_\{i,:\}\\in\\mathbb\{R\}^\{L\}is a vector of PL\_Alpha\_Hill of modules of the same type\. For a standard transformer architecture, we haveNmod=7N\_\{\\text\{mod\}\}=7\. The tensor representation of spectral signature creates a mappingΘ→ℝNmod×L\\Theta\\to\\mathbb\{R\}^\{N\_\{\\text\{mod\}\}\\times L\}that extracts spectral information of each weight matrix in the model\. It allows module\-level analysis, as different types of transformer layers often undergo different degrees of structural changes during post\-training\. Our empirical observation shows that Multi\-Layer Perceptron \(MLP\) projection layers present larger separation in independently trained models compared to attention layers\.

### 4\.2\.Spectral Signatures for Model Quantification

The spectral signature encodes robust spectral information of each weight matrix in an LLM, which can be used for systematic management and quantification of large collections of LLMs\. In this section, we describe how spectral signatures can be systematically applied to model quantification tasks, and we demonstrate their use in the following representative scenarios\.

#### Measuring LLM Similarities\.

To quantify the relations between models in a model collectionℱ\\mathcal\{F\}, we define a similarity measure based on their spectral signatures\. Let𝐙\(k\)∈ℝNmod×L\\mathbf\{Z\}^\{\(k\)\}\\in\\mathbb\{R\}^\{N\_\{\\text\{mod\}\}\\times L\}denote the spectral signature of modelkk, where theii\-th row𝐙i,:\(k\)∈ℝL\\mathbf\{Z\}^\{\(k\)\}\_\{i,:\}\\in\\mathbb\{R\}^\{L\}corresponds to the vector of PL\_Alpha\_Hill of all occurrences of theii\-th module type \(e\.g\., Attention Q/K/V, MLP\)\. Given two modelsfθ1,fθ2f\_\{\\theta\_\{1\}\},f\_\{\\theta\_\{2\}\}with signatures𝐙\(1\),𝐙\(2\)\\mathbf\{Z\}^\{\(1\)\},\\mathbf\{Z\}^\{\(2\)\}, we compute the pairwise similarity by averaging the Spearman rank correlations across aligned modules:

\(2\)Sim​\(𝐙\(1\),𝐙\(2\)\)=1Nmod​∑i=1Nmodρs​\(𝐙i,:\(1\),𝐙i,:\(2\)\),\\mathrm\{Sim\}\(\\mathbf\{Z\}^\{\(1\)\},\\mathbf\{Z\}^\{\(2\)\}\)=\\frac\{1\}\{N\_\{\\text\{mod\}\}\}\\sum\_\{i=1\}^\{N\_\{\\text\{mod\}\}\}\\rho\_\{s\}\(\\mathbf\{Z\}^\{\(1\)\}\_\{i,:\},\\mathbf\{Z\}^\{\(2\)\}\_\{i,:\}\),whereρs​\(⋅,⋅\)\\rho\_\{s\}\(\\cdot,\\cdot\)is the Spearman’s rank correlation coefficient:

\(3\)ρs​\(𝐮,𝐯\)=corr​\(rank​\(𝐮\),rank​\(𝐯\)\)\.\\rho\_\{s\}\(\\mathbf\{u\},\\mathbf\{v\}\)=\\mathrm\{corr\}\\\!\\left\(\\mathrm\{rank\}\(\\mathbf\{u\}\),~\\mathrm\{rank\}\(\\mathbf\{v\}\)\\right\)\.We use Spearman correlation in Eqs\.[2](https://arxiv.org/html/2607.03377#S4.E2)–[3](https://arxiv.org/html/2607.03377#S4.E3)because it depends only on the relative depth\-wise ordering of PL\_Alpha\_Hill values, making the similarity less sensitive to module\-specific affine distortions of the statistics and improving comparability across scales\. More importantly, PL\_Alpha\_Hill itself provides exact invariances that directly match our output\-invariant stress tests\. For any weight matrix𝐖\\mathbf\{W\}, let𝐂=𝐖⊤​𝐖\\mathbf\{C\}=\\mathbf\{W\}^\{\\top\}\\mathbf\{W\}with eigenvalues\{λj\}\\\{\\lambda\_\{j\}\\\}\. If𝐖′=c​𝐏𝐖𝐐\\mathbf\{W\}^\{\\prime\}=c\\,\\mathbf\{P\}\\mathbf\{W\}\\mathbf\{Q\}wherec\>0c\>0and𝐏,𝐐\\mathbf\{P\},\\mathbf\{Q\}are orthogonal \(including permutations\), then𝐖′⊤​𝐖′=c2​𝐐⊤​𝐂𝐐\{\\mathbf\{W\}^\{\\prime\}\}^\{\\\!\\top\}\\mathbf\{W\}^\{\\prime\}=c^\{2\}\\,\\mathbf\{Q\}^\{\\top\}\\mathbf\{C\}\\mathbf\{Q\}has eigenvaluesλj′=c2​λj\\lambda^\{\\prime\}\_\{j\}=c^\{2\}\\lambda\_\{j\}\. Since the Hill estimator used for PL\_Alpha\_Hill depends on log\-ratios of tail eigenvalues, these ratios are unchanged and thusP​L​\_​A​l​p​h​a​\_​H​i​l​l​\(𝐖′\)=P​L​\_​A​l​p​h​a​\_​H​i​l​l​\(𝐖\)PL\\\_\\allowbreak Alpha\\\_\\allowbreak Hill\(\\mathbf\{W\}^\{\\prime\}\)=PL\\\_\\allowbreak Alpha\\\_\\allowbreak Hill\(\\mathbf\{W\}\)\. This yields permutation\- and scale\-robust similarity without solving explicit neuron\-matching problems that arise in matrix\-level fingerprints such asAWM\(Zenget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib42)\)\.

#### Model Classification

The properties of the spectral signature further enable its use for identifying the lineage of an LLM among candidate base models, as models derived from the same base model exhibit highly similar eigenspectrum shapes, whereas models with different origins show substantial divergence in shape\. We approach the lineage identification as an unsupervised classification problem in the spectral signature space, and we use a nearest\-neighbor classifier based on the spectral similarity from Eq\.[2](https://arxiv.org/html/2607.03377#S4.E2)\. Let\{𝐙\(k\)\}k=1K\\\{\\mathbf\{Z\}^\{\(k\)\}\\\}\_\{k=1\}^\{K\}denote the set of spectral signatures forKKknown base foundation models\. For a given query model with spectral signatureZZ, we compute its similarity to each base model via the spectral signature similarity formula defined in Eq\.[2](https://arxiv.org/html/2607.03377#S4.E2)\. The predicted lineagek^\\hat\{k\}is the base model that maximizes this similarity:

\(4\)k^=arg​maxk∈\{1,…,K\}⁡Sim⁡\(𝐙,𝐙\(k\)\)\\hat\{k\}=\\operatorname\*\{arg\\,max\}\_\{k\\in\\\{1,\\dots,K\\\}\}\\operatorname\{Sim\}\(\\mathbf\{Z\},\\mathbf\{Z\}^\{\(k\)\}\)

#### Model Clustering\.

Spectral signature of LLMs further enables the discovery of underlying genealogical structures within large model collections without supervision\. We formulate this problem as a clustering problem in the space of spectral signatures\. Given a model collectionℱ=\{fθ1,…,fθN\}\\mathcal\{F\}=\\\{f\_\{\\theta\_\{1\}\},\\dots,f\_\{\\theta\_\{N\}\}\\\}, we extract the spectral signature𝐙\(k\)∈ℝNmod×L\\mathbf\{Z\}^\{\(k\)\}\\in\\mathbb\{R\}^\{N\_\{\\text\{mod\}\}\\times L\}of each modelfθkf\_\{\\theta\_\{k\}\}\. For vector\-based clustering algorithms, we flatten𝐙\(k\)\\mathbf\{Z\}^\{\(k\)\}into a feature vector𝐳\(k\)∈ℝNmod⋅L\\mathbf\{z\}^\{\(k\)\}\\in\\mathbb\{R\}^\{N\_\{\\text\{mod\}\}\\cdot L\}\. We use spectral signatures for four unsupervised pipelines: HDBSCAN, Bayesian Gaussian Mixture Models \(BGMM\), Spectral Clustering, and K\-Means\. Due to the high dimensionality of the spectral signature representations, we apply Uniform Manifold Approximation and Projection \(UMAP\) as a preprocessing step for HDBSCAN, BGMM, and K\-Means\. In contrast, Spectral Clustering operates directly on the original high\-dimensional pairwise distance matrix, where dimensionality reduction is implicitly handled through the graph Laplacian eigenstructure\. Hyperparameters for clustering algorithms \(i\.e\., number of clusters\) are selected by maximizing the silhouette score\. To ensure that clustering quality reflects intrinsic structure in the weight space rather than artifacts introduced by the embedding process, all internal evaluation metrics \(e\.g\., silhouette scores\) are computed using distances in the original spectral signature space\. We assess the robustness and generalizability of each clustering pipeline by repeating experiments under random subsampling at varying sample sizes, verifying consistent clustering behavior across regimes\.

#### Predicting LLM Performance\.

To quantify the predictive power of the proposed spectral signature, we treat the performance estimation as a non\-parametric regression task over the model manifold\. Given a query modelfθf\_\{\\theta\}with spectral signature𝐙\\mathbf\{Z\}and a reference pool of models𝒫=\{\(𝐙\(i\),𝐛i\)\}i=1N\\mathcal\{P\}=\\\{\(\\mathbf\{Z\}^\{\(i\)\},\\mathbf\{b\}\_\{i\}\)\\\}\_\{i=1\}^\{N\}, where𝐙\(i\)\\mathbf\{Z\}^\{\(i\)\}is the spectral signature and𝐛i\\mathbf\{b\}\_\{i\}is the vector of benchmark scores, we estimate the performance offθf\_\{\\theta\}using distance\-weightedkk\-nearest neighbor \(kk\-NN\) Interpolation\. Specifically, we first derive a distance metricd​\(𝐙,𝐙\(i\)\)d\(\\mathbf\{Z\},\\mathbf\{Z\}^\{\(i\)\}\)from the spectral similaritySim​\(𝐙,𝐙\(i\)\)\\mathrm\{Sim\}\(\\mathbf\{Z\},\\mathbf\{Z\}^\{\(i\)\}\)\. We employ the transformationd​\(𝐙,𝐙\(i\)\)=1−Sim​\(𝐙,𝐙\(i\)\)d\(\\mathbf\{Z\},\\mathbf\{Z\}^\{\(i\)\}\)=1\-\\mathrm\{Sim\}\(\\mathbf\{Z\},\\mathbf\{Z\}^\{\(i\)\}\)\. This ensures that models with highly correlated spectral shapes are positioned closer in the embedding space\. The predicted benchmark scores𝐛~\\tilde\{\\mathbf\{b\}\}are computed by aggregating the top\-kkmost similar neighbors in𝒫\\mathcal\{P\}:

𝐛~=∑i∈𝒦wi⋅𝐛i∑i∈𝒦wi,\\tilde\{\\mathbf\{b\}\}=\\frac\{\\sum\_\{i\\in\\mathcal\{K\}\}w\_\{i\}\\cdot\\mathbf\{b\}\_\{i\}\}\{\\sum\_\{i\\in\\mathcal\{K\}\}w\_\{i\}\},, where𝒦\\mathcal\{K\}is the set ofkknearest neighbors, andwi=1/\(d​\(𝐙,𝐙\(i\)\)\+ϵ\)w\_\{i\}=1/\(d\(\\mathbf\{Z\},\\mathbf\{Z\}^\{\(i\)\}\)\+\\epsilon\)is the assigned weight\. This interpolation relies on the assumption that models that exhibit similar HT\-SR patterns in their weight matrices are likely to possess comparable generalization capabilities\. We then quantify the prediction accuracy with:

\(5\)Dp​e​r​f=‖𝐛~−𝐛‖,D\_\{perf\}=\\\|\\tilde\{\\mathbf\{b\}\}\-\\mathbf\{b\}\\\|,where𝐛\\mathbf\{b\}represents the ground truth benchmark scores\. Typically,∥⋅∥\\\|\\cdot\\\|in Eq\.[5](https://arxiv.org/html/2607.03377#S4.E5)is defined by mean absolute error \(MAE\)\.

## 5\.Experiments

In this section, we demonstrate that spectral signatures provide an efficient and accurate tool for a range of downstream tasks\. Specifically, we present details of the experimental setup as well as the results of \(i\) measuring LLM similarities, \(ii\) model classification and clustering, and \(iii\) predicting and ranking model performance\.

### 5\.1\.Measuring LLM Similarities

#### Model Corpus\.

We select Llama\-2\-7b\(Touvronet al\.,[2023b](https://arxiv.org/html/2607.03377#bib.bib58)\)as the base model and compare it with four types of related variants that share the same architecture: \(i\) post\-trained models \(SFT and RL\), \(ii\) merged models and pruned models, \(iii\) noise\-perturbed models obtained by adding i\.i\.d\. Gaussian perturbations to all weight entries with noise strengthγ∈\{0\.05,0\.30,1\.00\}\\gamma\\in\\\{0\.05,0\.30,1\.00\\\}, as defined in Appendix[7\.6](https://arxiv.org/html/2607.03377#S7.SS6), \(iv\) output\-invariant transformations on adjacent matrices and output\-invariant permutations of hidden units, as defined in Appendix[7\.6](https://arxiv.org/html/2607.03377#S7.SS6)\. This corpus isolates a key practical goal: a lineage/similarity signal should remain stable under post\-training and output\-invariant reparameterizations, while being informative under destructive edits \(e\.g\., aggressive pruning\)\. We further validate the generalizability of our spectral signature to Mixture\-of\-Experts architectures and cross\-depth model pairs in Appendix[7\.5](https://arxiv.org/html/2607.03377#S7.SS5)

#### Setup\.

We compare against both*data\-aware*and*data\-free*baselines\. For data\-aware methods, we useREEF\(Zhanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib48)\), which computes similarity from activation features on a fixed prompt set \(300 prompts for TruthfulQA\(Linet al\.,[2022](https://arxiv.org/html/2607.03377#bib.bib52)\)\), andLogits\(Yang and Wu,[2024](https://arxiv.org/html/2607.03377#bib.bib53)\), which measures similarity from model outputs on the same prompts\. For data\-free approaches, we applyPCS\(Zenget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib3)\)that computes cosine similarity between flattened weight tensors under a fixed alignment, and discuss alignment\-based weight signature \(e\.g\.,AWM\(Zenget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib42)\)\) as a complementary point of comparison: these methods operate at the matrix level and often require per\-pair matching/alignment, which makes them less convenient as a drop\-in kernel when building dense similarity matrices for zoo\-scale clustering and retrieval\.

#### Results\.

Tab\.[2](https://arxiv.org/html/2607.03377#S5.T2)summarizes the overall similarity performance of our method compared to baseline models\. It demonstrates the robustness of our spectral signature under diverse scenarios, including fine\-tuning, pruning, model merging, noise injection, and output\-invariant attacks\. First, the similarity scores of our method remain consistently high among Llama\-2\-7b related variants, such as multiple fine\-tuned models, with scores in the\[0\.97,1\.0\]\[0\.97,1\.0\]range across instruction tuning and domain shifts\. In contrast, data\-aware baselines can fail under prompt\- or language\-induced behavior changes\. For example, Logits drops sharply for the Chinese variants, and REEF degrades for the LoRA\-unloaded model, whereas our scores remain stable\. Under weight noise injection, our similarity decreases mildly asσ\\sigmaincreases from≈0\.999\\approx 0\.999atγ=0\.05\\gamma=0\.05to≈0\.979\\approx 0\.979atγ=1\.00\\gamma=1\.00, outperforming PCS and REEF at the strongest noise perturbation\. Moreover, our method is robust to hidden\-unit permutations and random scaling under output\-invariant transformations, while PCS is sensitive to both\. For unstructured pruning, the similarity score using spectral signature decreases gradually at 30–50% sparsity and drops substantially at 70% sparsity, indicating that the spectral signature is informative of the structural changes under more aggressive parameter removal\.

Quantifying similarity among large model corpora typically requires constructing a stable and efficient pairwise similarity matrix\. However, existing methods often fall short computationally: data\-dependent methods require prompts and incur inference costs for each comparison, while pair\-specific methods such asAWMincur cumulative overhead when constructing dense similarity matrices over large model collections\. Our approach computes a compact signature once per model and yields a computationally\-cheap similarity that remains robust under the varying transformations that break coordinate\-dependent weight distances, making it practical for large\-scale lineage analysis\.

Table 2\.A comprehensive evaluation of signature\-based methods for measuring LLM similarity across diverse settings\. Here,,, anddenote similarity scores\>0\.8\>0\.8\(high\), between0\.50\.5and0\.80\.8\(moderate\), and<0\.5<0\.5\(low\), respectively\.Model Fine\-tuningllama2\-7b\-hf\-instruction\-loravicuna\-\-v1\.5Nous\-Hermes\-llama\-2\-7bchinese\-llama\-2\-7bLlama\-2\-7b\-hf\-instruct\-pl\-lora\_unloadchinese\-alpaca\-2\-7bPCS\(Zenget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib3)\)0\.990\.990\.990\.990\.990\.990\.930\.930\.990\.990\.950\.95Logits\(Yang and Wu,[2024](https://arxiv.org/html/2607.03377#bib.bib53)\)0\.980\.980\.790\.790\.990\.990\.030\.030\.990\.990\.560\.56REEF\(Zhanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib48)\)0\.770\.770\.990\.990\.990\.990\.990\.990\.140\.140\.990\.99GhostSpec\(Wanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib41)\)0\.980\.980\.990\.990\.990\.990\.800\.800\.870\.870\.830\.83Ours0\.990\.990\.990\.990\.990\.990\.970\.970\.990\.990\.970\.97Unrelated Modelsllama\-7bllama\-3\.1\-8BMistral\-7BQwen\-1\.5\-7Bopen\-llama\-7bopen\-llama\-7b\-v2PCS\(Zenget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib3)\)0\.010\.010\.140\.140\.270\.270\.060\.060\.010\.010\.010\.01Logits\(Yang and Wu,[2024](https://arxiv.org/html/2607.03377#bib.bib53)\)0\.990\.990\.000\.000\.770\.770\.000\.000\.960\.960\.950\.95REEF\(Zhanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib48)\)0\.150\.150\.150\.150\.130\.130\.140\.140\.170\.170\.180\.18GhostSpec\(Wanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib41)\)0\.940\.940\.190\.190\.000\.000\.890\.890\.910\.910\.800\.80Ours0\.920\.920\.660\.660\.770\.770\.680\.680\.720\.720\.600\.60Unstructured PruningRL Models30%50%70%Archer\-Code\-1\.5B \(GRPO\)Nemotron\-Resear\-ch\-Reasoning\-Qwen\-1\.5B\(Reinforcement\+\+\)Polaris\-7B\-Preview\(DAPO\)PCS\(Zenget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib3)\)0\.990\.990\.900\.900\.780\.780\.990\.990\.990\.991\.001\.00Logits\(Yang and Wu,[2024](https://arxiv.org/html/2607.03377#bib.bib53)\)1\.001\.001\.001\.000\.990\.991\.001\.001\.001\.001\.001\.00REEF\(Zhanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib48)\)0\.990\.990\.980\.980\.980\.980\.980\.980\.960\.960\.940\.94GhostSpec\(Wanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib41)\)0\.990\.990\.890\.890\.700\.701\.001\.001\.001\.000\.460\.46Ours0\.890\.890\.890\.890\.400\.400\.990\.990\.990\.990\.990\.99Noise InjectionWeight Merging \(Evollm\-jp\)γ=0\.05\\gamma=0\.05γ=0\.30\\gamma=0\.30γ=1\.00\\gamma=1\.00shisa\-gamma\-7bwizardmath\-7b\-1\.1abel\-7b\-002PCS\(Zenget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib3)\)0\.990\.990\.950\.950\.700\.700\.990\.990\.990\.990\.990\.99Logits\(Yang and Wu,[2024](https://arxiv.org/html/2607.03377#bib.bib53)\)1\.001\.001\.001\.000\.970\.970\.000\.001\.001\.001\.001\.00REEF\(Zhanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib48)\)0\.990\.990\.990\.990\.690\.690\.930\.930\.910\.910\.890\.89GhostSpec\(Wanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib41)\)1\.001\.000\.970\.970\.700\.700\.230\.230\.990\.990\.990\.99Ours0\.990\.990\.980\.980\.970\.970\.800\.800\.870\.870\.830\.83

### 5\.2\.Model Classification and Clustering

#### Model Corpus\.

We construct a corpus of 128 publicly available models derived from 8 model families: llama\-7b\(Touvronet al\.,[2023a](https://arxiv.org/html/2607.03377#bib.bib57)\), llama\-2\-7b\(Touvronet al\.,[2023b](https://arxiv.org/html/2607.03377#bib.bib58)\), llama\-3\.1\-8b\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib59)\), mistral\-7b\-v0\.1, mistral\-7b\-v0\.2\(Jianget al\.,[2023](https://arxiv.org/html/2607.03377#bib.bib56)\), open\-llama\-7b, open\-llama\-7b\-v2\(Geng and Liu,[2023](https://arxiv.org/html/2607.03377#bib.bib54)\), and qwen\-1\.5\-7b\(Baiet al\.,[2023](https://arxiv.org/html/2607.03377#bib.bib55)\)\. For each derived model, we assign a training\-lineage label corresponding to one of the eight base models, using the model’s publicly documented training provenance\. For the classification task, we classify each of the 128 derived models into one of the 8 base families by selecting the base with the highest spectral signature similarity score to the model\. For the clustering task, we perform unsupervised clustering on the signatures of all 128 derived models and 8 base models\.

#### Setup\.

For the classification task, we compute the similarity score using three different kernels on spectral signatures: Spearman rank correlation, radial basis function \(RBF\) kernel, and cosine similarity \(detailed in Appendix Sec\.[7\.2](https://arxiv.org/html/2607.03377#S7.SS2)\)\. This allows us to assess the robustness of classification performance under different similarity measures\. For the clustering task, we conduct two complementary studies\. First, we assess clustering performance using spectral signatures constructed with different ESD\-based metrics, including shape metrics \(PL\_Alpha\_Hill\), scale metrics \(Log\_Norm and Spectral\_Norm\), and a combined shape–scale metric \(Log\_Alpha\_Norm\)\. Second, we evaluate the robustness of clustering by applying four clustering algorithms, as described in Sec\.[4](https://arxiv.org/html/2607.03377#S4), to these spectral signatures\. We assess clustering quality using the Silhouette Coefficient\(Rousseeuw,[1987](https://arxiv.org/html/2607.03377#bib.bib50)\)and the Davies–Bouldin Index\(Davies and Bouldin,[2009](https://arxiv.org/html/2607.03377#bib.bib51)\), as detailed in Appendix[7\.3](https://arxiv.org/html/2607.03377#S7.SS3)\.

#### Classification Results\.

Fig\.[3](https://arxiv.org/html/2607.03377#S5.F3)shows that the proposed spectral signature enables accurate lineage classification across all eight model families with an overall accuracy of 98\.44%\. Across similarity kernels, the classifier consistently achieves near\-perfect accuracy, indicating that lineage information is strongly preserved in the spectral signature\. Performance is stable across Spearman rank, RBF, and cosine similarity, suggesting that the results are not sensitive to the choice of similarity measure\. Minor accuracy variations are mainly observed for closely related families, such as mistral\-7b\-v0\.1 and mistral\-7b\-v0\.2, which correspond to different versions within the same model line and share highly similar architectures and training procedures\. Overall, these results demonstrate that spectral signatures provide a reliable and discriminative basis for model lineage classification\.

#### Clustering Results\.

Tab\.[3](https://arxiv.org/html/2607.03377#S5.T3)compares the clustering performance of spectral signatures constructed from scale\- and shape\-based metrics\. The proposed shape\-based PL\_Alpha\_Hill achieves the best performance, with the highest Silhouette score of 0\.91 and the lowest Davies\-Bouldin index of 0\.22, consistently outperforming the scale\-based alternatives\. To further evaluate the robustness of our proposed spectral signature, in Tab\.[4](https://arxiv.org/html/2607.03377#S5.T4)we report the result of different clustering algorithms\. All clustering algorithms that use the spectral signature achieve high silhouette scores, ranging from 0\.73 to 0\.91\. This shows that the proposed spectral signature provides a robust and effective embedding for different classes of clustering algorithms\. As an illustrative example, in Fig\.[9](https://arxiv.org/html/2607.03377#S7.F9)we provide the clustering result of 2D t\-SNE using the spectral clustering algorithm with spectral signatures constructed by PL\_Alpha\_Hill\. As we can see, the proposed spectral signature clearly separates different model families, resulting in distinct clusters\.

![Refer to caption](https://arxiv.org/html/2607.03377v1/x3.png)Figure 3\.Classification accuracy of the 8\-way classifier for routing derived models to their base models, reported across model families and similarity kernels\. Each bar shows accuracy for one family\-kernel pair\.Table 3\.Comparison of scale\- and shape\-based spectral signatures for spectral clustering\.MetricSilhouette \(↑\)Davies\_Bouldin \(↓\)Log\_Norm \(scale\)0\.700\.47Spectral\_Norm \(scale\)0\.810\.35Log\_Alpha\_Norm \(scale\)0\.880\.23PL\_Alpha\_Hill \(shape\)0\.910\.22

Table 4\.Comparing different clustering algorithms using the spectral signature constructed by PL\_Alpha\_Hill\.MethodSilhouette \(↑\\uparrow\)Davies\_Bouldin \(↓\\downarrow\)HDBSCAN0\.730\.56BGMM0\.730\.56K\-Means0\.730\.56Spectral Clustering0\.910\.22

### 5\.3\.Quantifying and Ranking LLM Performance

#### Model Corpus\.

For model ranking and prediction tasks, we curate 499 open\-source models from the Open LLM Leaderboard\(Myrzakhanet al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib43)\)\. This corpus spans a broad spectrum of architectures, focusing on the Llama, Mistral, GPT, and Pythia families, with parameter scales ranging from 19 million to 70 billion\. Further validation on additional scales and intra\-family prediction is provided in Appendix[7\.5](https://arxiv.org/html/2607.03377#S7.SS5)\.

#### Setup\.

To evaluate whether spectral signatures can predict downstream performance, we consider four representative benchmarks: ARC\-Challenge \(reasoning\)\(Clarket al\.,[2018](https://arxiv.org/html/2607.03377#bib.bib46)\), HellaSwag \(commonsense\)\(Zellerset al\.,[2019](https://arxiv.org/html/2607.03377#bib.bib45)\), MMLU \(world knowledge\)\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.03377#bib.bib44)\), and TruthfulQA \(factuality\)\(Linet al\.,[2022](https://arxiv.org/html/2607.03377#bib.bib52)\)\. The models span a wide performance range \(e\.g\., MMLU scores from 23\.0 to 70\.67\), covering both low\- and high\-performance regimes\. For each model, we estimate performance using distance\-weightedkk\-NN interpolation \(k=3k=3\) based on spectral\-signature similarity\. We use 5\-fold cross\-validation to assess generalization\.

We compare against two activation\-based baselines\. First of all, EmbedLLM\(Zhuanget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib28)\)encodes each model by extracting the hidden\-state embedding at the final token position from the last transformer layer on a fixed prompt set𝒫\\mathcal\{P\}, and trains a linear regressor to predict benchmark scores under leave\-one\-out evaluation\. Second, LLMDNA\(Wuet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib40)\)extracts a128×128128\\times 128model representation from 100 random samples, which is nearly invariant to the specific dataset or questions used for extraction\. We use the same regression protocol for LLMDNA as for EmbedLLM\. Prediction quality is evaluated by MAE and Kendall’s Tau rank correlation\(Kendall,[1948](https://arxiv.org/html/2607.03377#bib.bib49)\)\. We define “Significance” as the number of repeated train/test splits \(out of 100\), where Kendall’s Tau between predicted and ground\-truth rankings is statistically significant at the 5% level\.

We further discuss why our spectral signatures are well suited for this setting\. Activation\-based performance quantification requires task data at inference, and does not scale well to frequent re\-ranking across large model collections\. Conversely, other weight\-based methods such asPCS\(Zenget al\.,[2024](https://arxiv.org/html/2607.03377#bib.bib3)\)andAWM\(Zenget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib42)\)are primarily designed for pair\-wise provenance checks and do not naturally provide a compact, geometry\-aware representation amenable to regression across heterogeneous architectures\. Our spectral signatures provide a lightweight, task\-agnostic representation that supports nearest\-neighbor interpolation without any prompts\.

#### Results\.

We run 100 repeated trials with random train/test splits \(249 models for training and 250 models for testing in each trial\)\. Fig\.[4](https://arxiv.org/html/2607.03377#S5.F4)compares predicted benchmark scores with their ground\-truth values\. The points cluster closely around the diagonal \(y=xy=x\), indicating good agreement between predicted and actual performance\. We report \(i\) the average MAE over the 100 trials, which measures absolute prediction error, and \(ii\) the number of trials in which Kendall’s Tau correlation between predicted and ground\-truth rankings on the test set is statistically significant\. As shown in Tab\.[5](https://arxiv.org/html/2607.03377#S5.T5), spectral signatures achieve consistently low MAE and statistically significant rank correlation in all trials across all benchmarks\. While EmbedLLM achieves slightly lower MAE on ARC and TruthfulQA, spectral signatures outperform EmbedLLM on MMLU and remain competitive on HellaSwag\. This gap reflects that benchmarks targeting specific model capabilities \(TruthfulQA, ARC\) are more directly reflected in task\-specific activations, while benchmarks covering general knowledge \(MMLU, HellaSwag\) align more closely with the pretraining capacity encoded in weight geometry\. LLMDNA\(Wuet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib40)\), despite requiring comparable inference cost, yields substantially higher MAE across all benchmarks\. Unlike EmbedLLM, which extracts task\-conditioned activations, LLMDNA produces a single task\-agnostic embedding per model\. The compression loses the benchmark\-specific variability needed for accurate performance prediction\.

![Refer to caption](https://arxiv.org/html/2607.03377v1/x4.png)Figure 4\.Predicted versus ground\-truth benchmark scores on MMLU and ARC across the model pool\. The dashed diagonal line denotes perfect agreement\.
#### Computational Cost\.

We report a runtime comparison between our spectral signature and baselines under the same model and hardware settings\. We use Llama\-2\-7b as the representative backbone and run all methods on a single NVIDIA L40 GPU with FP16 precision\. For EmbedLLM, we follow the original pipeline and extract last\-token hidden\-state embeddings on a fixed prompt set\. For LLMDNA, we largely follow the original pipeline, with slight modifications to adjust for our hardware and specific models in our dataset\. For our method, we compute the spectral signatures directly from model weights without any forward passes\. We measureRun Timefrom the end of model loading to the end of signature/embedding extraction\. One iteration of EmbedLLM embeddings takes 2,633 seconds, and one iteration of LLMDNA embeddings takes 5,393 seconds, while the Runtime of Spectral Signature is 337 seconds, significantly more time\-efficient than the baselines\.

Table 5\.Performance prediction results on four benchmarks, comparing our spectral signatures, EmbedLLM, and LLMDNA using MAE and statistical significance \(Signif\.\)\.OursEmbedLLMLLMDNABenchmarkMAESignif\.MAESignif\.MAESignif\.ARC2\.271001\.611007\.12100HellaSwag3\.251003\.5010011\.65100MMLU1\.951003\.141006\.78100TruthfulQA2\.991001\.961003\.43100

### 5\.4\.Generalization to Diverse Architectures

Here, we test our spectral signature on more diverse architectures\.

#### Model Lineage on MoE models

We evaluate our spectral signature on Pangu\-Pro\-MoE\(Tanget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib60)\), an MoE architecture derived from the Qwen2\.5\-14B family\. To construct spectral signatures for MoE modules, we average the spectral metrics across experts\. Tab\.[6](https://arxiv.org/html/2607.03377#S5.T6)reports the similarity between Pangu\-Pro\-MoE and candidate source models\. Our method identifies Qwen2\.5\-14B, Qwen2\.5\-Coder\-14B, and OpenMath\-Nemotron\-14B as the most likely sources, demonstrating that spectral signatures generalize to MoE architectures with minimal adaptation\.

Table 6\.Similarities between Pangu\-Pro\-MoE and candidate source models\.Candidate ModelSimilarityQwen/Qwen2\.5\-14B0\.91nvidia/OpenMath\-Nemotron\-14B0\.87Qwen/Qwen2\.5\-Coder\-14B0\.84Qwen/Qwen1\.5\-14B0\.3001\-ai/Yi\-1\.5\-9B0\.07meta\-llama/Llama\-2\-13b\-hf0\.04

#### Similarity Between Depth\-Mismatched Models\.

Models derived from the same source may differ in depth\. We use an order\-preserving layer matching via dynamic programming \(DP\)\(Bellman,[1958](https://arxiv.org/html/2607.03377#bib.bib61)\)on spectral signatures\. We build a cost matrixD​\[i,j\]=m​e​a​n​\|Z1​\[i\]−Z2​\[j\]\|D\[i,j\]=mean\|Z\_\{1\}\[i\]\-Z\_\{2\}\[j\]\|over 7 z\-score normalized module columns, then find the injective order\-preserving mapping minimizing total cost via DP\. We evaluate on the model pair \(Llama\-3\.1\-8B vs\. Llama\-3\.2\-3B\) to enable direct comparison with their activation\-based ground truth\. Tab\.[7](https://arxiv.org/html/2607.03377#S5.T7)shows a3\.7×3\.7\\timeslower matching cost for the related pair\. Moreover, the optimal matching skips layers \{11, 17, 23, 29\} of Llama\-3\.1\-8B, consistent with prior activation\-based matching results\(Zhuet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib29)\)\. This agreement suggests that our data\-free spectral signatures capture structural information for cross\-depth layer alignment\.

Table 7\.Matching cost and post\-matching Spearman correlation\.Candidate ModelMatching cost \(↓\\downarrow\)Similarity \(↑\\uparrow\)Llama\-3\.2\-3B0\.260\.89Qwen2\.5\-7B0\.950\.77

### 5\.5\.Ablation Study

PL\_Alpha\_Hill remains robust under output\-invariant reparameterizations\.We evaluate whether PL\_Alpha\_Hill is a robust metric for constructing a spectral signature by performing output\-invariant reparameterizations on weight matrices before measuring model similarity with Eq\.[2](https://arxiv.org/html/2607.03377#S4.E2)\. We compare the spectral signature constructed by PL\_Alpha\_Hill with other ESD scale metrics \(Log Norm and Spectral Norm\)\. Concretely, we apply two reparameterizations on LLaMA\-2\-7B model: \(i\) using output\-invariant scaling to consecutive attention modules \(Q/K and V/O pairs\), and \(ii\) output\-invariant change of basis between two consecutive linear layers in the feed\-forward network \(FFN\) module \(more details can be found in Appendix[7\.6](https://arxiv.org/html/2607.03377#S7.SS6)\)\. From Fig\.[5](https://arxiv.org/html/2607.03377#S5.F5), we can see that applying output\-invariant weight scaling causes PCS\-based and ESD scale\-based similarity to degrade substantially as we use larger scaling factors, while the spectral signature with PL\_Alpha\_Hill remains stable, correctly recognizing the transformed weights as functionally identical\. Furthermore, under MLP hidden\-unit permutations, PCS collapses to near zero, but PL\_Alpha\_Hill remains near\-perfect \(Tab\.[8](https://arxiv.org/html/2607.03377#S5.T8)\)\.

Here, we discuss why this ablation is important\. Data\-aware similarity \(e\.g\., Logits and REEF\) is expected to remain high under strictly output\-invariant transforms, but it requires a prompt distribution andO​\(\#​prompts\)O\(\\\#\\text\{prompts\}\)inference per comparison, making it unsuitable as a zoo\-scale kernel\. Weight\-matrix fingerprints such asAWM\(Zenget al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib42)\)reduce some reparameterization sensitivity but still require matrix\-level comparisons \(and often assignment/matching\) per pair\. In contrast, our ESD shape signature is computed once per model and yields invariance to common output\-preserving symmetries without explicit matching\. Note that pure scale metrics \(Log Norm, Spectral Norm\) are also invariant here, but precisely because they track magnitude, they tend to saturate and are less discriminative across independently trained families \(see Sec\.[5\.1](https://arxiv.org/html/2607.03377#S5.SS1)\)\.

PL\_Alpha\_Hill remains robust under random perturbations\.To simulate post\-training weight changes, we inject Gaussian noise of increasing magnitude into model weight entries and evaluate the stability of different similarity metrics\. We follow the noise strengthγ\\gammadefinition in Appendix[7\.6](https://arxiv.org/html/2607.03377#S7.SS6)and report full results in Tab\.[9](https://arxiv.org/html/2607.03377#S5.T9)\. Across all noise levels, our PL\_Alpha\_Hill –based similarity remains close to the unperturbed model, but PCS degrades substantially as noise magnitude increases\. These results indicate that spectral shape features provide a robust measure of model similarity under realistic weight\-space perturbations\. Moreover, spectral scale metrics also perform well in this scenario since additive Gaussian noise perturbs weights without strongly changing their overall magnitude\. Beyond global averages, we also analyze the layer\-wise response to perturbation\. Fig\.[6](https://arxiv.org/html/2607.03377#S5.F6)demonstrates that larger perturbations generally increase PL\_Alpha\_Hill while preserving similar depth\-wise trends, showing that spectral signatures respond systematically to weight\-space noise\.

![Refer to caption](https://arxiv.org/html/2607.03377v1/x5.png)Figure 5\.Robustness of different similarity metrics under output\-invariant scaling to consecutive Query and Key modules\. PL\_Alpha\_Hill is a shape metric, while Log Norm and Spectral Norm are scale metrics\.Table 8\.Robustness of different metrics under hidden\-unit permutation\.MetricMLP\-InMLP\-OutPCS0\.000\.00Log\_Norm1\.001\.00Spectral\_Norm1\.001\.00Log\_Alpha\_Norm1\.001\.00PL\_Alpha\_Hill \(ours\)1\.001\.00

Table 9\.Similarities of distinct metrics under increasing noise\.Metric𝜸=0\.1\\boldsymbol\{\\gamma\}~\\mathbf\{=0\.1\}𝜸=0\.3\\boldsymbol\{\\gamma\}~\\mathbf\{=0\.3\}𝜸=0\.5\\boldsymbol\{\\gamma\}~\\mathbf\{=0\.5\}𝜸=1\.0\\boldsymbol\{\\gamma\}~\\mathbf\{=1\.0\}𝜸=2\.0\\boldsymbol\{\\gamma\}~\\mathbf\{=2\.0\}PCS1\.000\.960\.890\.700\.44Log\_Norm1\.001\.001\.001\.001\.00Spectral\_Norm1\.001\.001\.000\.980\.89Log\_Alpha\_Norm0\.990\.870\.850\.810\.81PL\_Alpha\_Hill \(ours\)1\.000\.990\.980\.980\.96

![Refer to caption](https://arxiv.org/html/2607.03377v1/x6.png)Figure 6\.Layer\-wise evolution of PL\_Alpha\_Hill under random perturbations\. Curves show the PL\_Alpha\_Hill across attention and MLP modules for different noise levelsγ\\gamma\.

## 6\.Conclusion

This work presents a spectral shape–based framework for large\-scale analysis of LLMs that is data\-free, computationally efficient, and robust to post\-training transformations\. We curate and benchmark a large and diverse LLM corpus and conduct extensive experiments across similarity measurement, classification, clustering, and performance prediction\. Our results have shown the consistent effectiveness of our spectral signature across various experimental setups\. Furthermore, we validate that ESD shape metrics, particularly PL\_Alpha\_Hill, capture intrinsic structural properties of LLMs that are stable under output\-invariant reparameterizations and noise perturbations\. Therefore, our spectral signature can provide a practical tool for managing modern LLM ecosystems, with implications for model provenance, organization, and scalable evaluation\.

###### Acknowledgements\.

YY would like to acknowledge the support of the U\.S\. Department of Energy \(Grant number: DE\-SC0025584\) and the Defense Advanced Research Projects Agency \(Grant number: HR00112520011\)\. HL was supported by the U\.S\. Department of Energy under Contract DE\-AC02\-05CH11231 and U\.S\. National Science Foundation NSF\-DMS 2412403\. We also would like to acknowledge NERSC DOE Mission Science Allocation for ERCAP request ERCAP0035003\.

## References

- J\. Alstott, E\. Bullmore, and D\. Plenz \(2014\)Powerlaw: a python package for analysis of heavy\-tailed distributions\.PloS one9\(1\),pp\. e85777\.Cited by:[footnote 1](https://arxiv.org/html/2607.03377#footnote1)\.
- J\. Bai, S\. Bai, Y\. Chu,et al\.\(2023\)Qwen technical report\.External Links:2309\.16609,[Link](https://arxiv.org/abs/2309.16609)Cited by:[§5\.2](https://arxiv.org/html/2607.03377#S5.SS2.SSS0.Px1.p1.1)\.
- R\. Bellman \(1958\)Dynamic programming\.Science127\(3304\),pp\. 976–976\.Cited by:[§5\.4](https://arxiv.org/html/2607.03377#S5.SS4.SSS0.Px2.p1.2)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.External Links:1803\.05457,[Link](https://arxiv.org/abs/1803.05457)Cited by:[§5\.3](https://arxiv.org/html/2607.03377#S5.SS3.SSS0.Px2.p1.2)\.
- A\. Clauset, C\. R\. Shalizi, and M\. E\. Newman \(2009\)Power\-law distributions in empirical data\.SIAM review51\(4\),pp\. 661–703\.Cited by:[footnote 1](https://arxiv.org/html/2607.03377#footnote1)\.
- R\. Couillet and Z\. Liao \(2022\)Random matrix methods for machine learning\.Cambridge University Press,Cambridge, UK\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1)\.
- D\. L\. Davies and D\. W\. Bouldin \(2009\)A cluster separation measure\.IEEE transactions on pattern analysis and machine intelligencePAMI\-1\(2\),pp\. 224–227\.Cited by:[§5\.2](https://arxiv.org/html/2607.03377#S5.SS2.SSS0.Px2.p1.1)\.
- X\. Geng and H\. Liu \(2023\)OpenLLaMA: an open reproduction of llamaOpenLM Research\.External Links:[Link](https://github.com/openlm-research/open_llama)Cited by:[§5\.2](https://arxiv.org/html/2607.03377#S5.SS2.SSS0.Px1.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri,et al\.\(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§5\.2](https://arxiv.org/html/2607.03377#S5.SS2.SSS0.Px1.p1.1)\.
- C\. Gu, C\. Huang, X\. Zheng, K\. Chang, and C\. Hsieh \(2022\)Watermarking pre\-trained language models with backdooring\.Vol\.abs/2210\.07543\.External Links:[Link](https://api.semanticscholar.org/CorpusID:252907247)Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Gueta, E\. Venezian, C\. Raffel, N\. Slonim, Y\. Katz, and L\. Choshen \(2023\)Knowledge is a region in weight space for finetuned language models\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Gurbuzbalaban, U\. Simsekli, and L\. Zhu \(2021\)The heavy\-tail phenomenon in sgd\.InInternational Conference on Machine Learning,Virtual,pp\. 3964–3975\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.External Links:2009\.03300,[Link](https://arxiv.org/abs/2009.03300)Cited by:[§5\.3](https://arxiv.org/html/2607.03377#S5.SS3.SSS0.Px2.p1.2)\.
- B\. M\. Hill \(1975\)A simple general approach to inference about the tail of a distribution\.The annals of statistics3\(5\),pp\. 1163–1174\.Cited by:[§3](https://arxiv.org/html/2607.03377#S3.SS0.SSS0.Px2.p2.1)\.
- L\. Hodgkinson and M\. Mahoney \(2021\)Multiplicative noise and heavy tails in stochastic optimization\.InInternational Conference on Machine Learning,Virtual,pp\. 4262–4274\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1)\.
- L\. Hodgkinson, U\. Simsekli, R\. Khanna, and M\. Mahoney \(2022\)Generalization bounds using lower tail exponents in stochastic optimizers\.InInternational Conference on Machine Learning,Baltimore, Maryland, USA,pp\. 8774–8795\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1)\.
- L\. Hodgkinson, Z\. Wang, and M\. W\. Mahoney \(2025\)Models of heavy\-tailed mechanistic universality\.Cited by:[footnote 1](https://arxiv.org/html/2607.03377#footnote1)\.
- E\. Horwitz, N\. Kurer, J\. Kahana, L\. Amar, and Y\. Hoshen \(2025a\)We should chart an atlas of all the world’s models\.Cited by:[§1](https://arxiv.org/html/2607.03377#S1.p1.1),[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px2.p1.1)\.
- E\. Horwitz, A\. Shul, and Y\. Hoshen \(2025b\)Unsupervised model tree heritage recovery\.InProceedings of the 13th International Conference on Learning Representations,Singapore,pp\. 47900–47918\.Cited by:[§1](https://arxiv.org/html/2607.03377#S1.p1.1),[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Hu, K\. Goel, V\. Killiakov, and Y\. Yang \(2025\)Eigenspectrum analysis of neural networks without aspect ratio bias\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research,Vancouver, Canada,pp\. 24290–24313\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1),[item 1](https://arxiv.org/html/2607.03377#S7.I1.i1.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch,et al\.\(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§5\.2](https://arxiv.org/html/2607.03377#S5.SS2.SSS0.Px1.p1.1)\.
- Y\. Jiang, C\. Wu, M\. K\. Boroujeny, B\. Mark, and K\. Zeng \(2025\)StealthInk: a multi\-bit and stealthy watermark for large language models\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research,Vancouver, Canada,pp\. 27685–27709\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Jin, C\. Zhang, S\. Shi, W\. Lou, and Y\. T\. Hou \(2024\)Proflingo: a fingerprinting\-based intellectual property protection scheme for large language models\.In2024 IEEE Conference on Communications and Network Security \(CNS\),Taipei, Taiwan,pp\. 1–9\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1)\.
- M\. G\. Kendall \(1948\)Rank correlation methods\.Charles Griffin & Company,London, UK\.Cited by:[§5\.3](https://arxiv.org/html/2607.03377#S5.SS3.SSS0.Px2.p2.2)\.
- Z\. Liao and M\. W\. Mahoney \(2025\)Random matrix theory for deep learning: beyond eigenvalues of linear models\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)Truthfulqa: measuring how models mimic human falsehoods\.InProceedings of the 60th annual meeting of the association for computational linguistics \(volume 1: long papers\),Dublin, Ireland,pp\. 3214–3252\.Cited by:[§5\.1](https://arxiv.org/html/2607.03377#S5.SS1.SSS0.Px2.p1.1),[§5\.3](https://arxiv.org/html/2607.03377#S5.SS3.SSS0.Px2.p1.2)\.
- Z\. Liu, Y\. Hu, T\. Pang, Y\. Zhou, P\. Ren, and Y\. Yang \(2024\)Model balancing helps low\-data training and fine\-tuning\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1311–1331\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.78/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.78)Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2607.03377#S3.SS0.SSS0.Px2.p2.5)\.
- H\. Lu, Y\. Zhou, S\. Liu, Z\. Wang, M\. W\. Mahoney, and Y\. Yang \(2024\)Alphapruning: using heavy\-tailed self regularization theory for improved layer\-wise pruning of large language models\.Advances in neural information processing systems37,pp\. 9117–9152\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Mahoney and C\. Martin \(2019\)Traditional and heavy tailed self regularization in neural network models\.InInternational Conference on Machine Learning,Long Beach, CA, USA,pp\. 4284–4293\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1)\.
- V\. A\. Marčenko and L\. A\. Pastur \(1967\)Distribution of eigenvalues for some sets of random matrices\.Mathematics of the USSR\-Sbornik1\(4\),pp\. 457–483\.Cited by:[item 1](https://arxiv.org/html/2607.03377#S7.I1.i1.p1.1)\.
- C\. H\. Martin and C\. Hinrichs \(2025\)SETOL: a semi\-empirical theory of \(deep\) learning\.Cited by:[§1](https://arxiv.org/html/2607.03377#S1.p2.1)\.
- C\. H\. Martin and M\. W\. Mahoney \(2021\)Implicit self\-regularization in deep neural networks: evidence from random matrix theory and implications for learning\.Journal of Machine Learning Research22\(165\),pp\. 1–73\.Cited by:[§1](https://arxiv.org/html/2607.03377#S1.p2.1),[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2607.03377#S3.SS0.SSS0.Px2.p1.5)\.
- C\. H\. Martin, T\. S\. Peng, and M\. W\. Mahoney \(2021\)Predicting trends in the quality of state\-of\-the\-art neural networks without access to training or testing data\.Nature Communications12\(1\),pp\. 1–13\.Cited by:[§1](https://arxiv.org/html/2607.03377#S1.p2.1)\.
- A\. Myrzakhan, S\. Mahmoud Bsharat, and Z\. Shen \(2024\)Open\-llm\-leaderboard: from multi\-choice to open\-style questions for llms evaluation, benchmark, and arena\.Cited by:[§5\.3](https://arxiv.org/html/2607.03377#S5.SS3.SSS0.Px1.p1.1)\.
- I\. Nikolic, T\. Baluta, and P\. Saxena \(2025\)Model provenance testing for large language models\.InAdvances in Neural Information Processing Systems,Sydney, Australia,pp\. 34126–34153\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1)\.
- P\. J\. Rousseeuw \(1987\)Silhouettes: a graphical aid to the interpretation and validation of cluster analysis\.Journal of computational and applied mathematics20,pp\. 53–65\.Cited by:[§5\.2](https://arxiv.org/html/2607.03377#S5.SS2.SSS0.Px2.p1.1)\.
- K\. Schürholt, L\. Meynent, Y\. Zhou, H\. Lu, Y\. Yang, and D\. Borth \(2025\)A model zoo on phase transitions in neural networks\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px2.p1.1)\.
- U\. Simsekli, L\. Sagun, and M\. Gurbuzbalaban \(2019\)A tail\-index analysis of stochastic gradient noise in deep neural networks\.InInternational Conference on Machine Learning,Long Beach, CA, USA,pp\. 5827–5837\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1)\.
- U\. Simsekli, O\. Sener, G\. Deligiannidis, and M\. A\. Erdogdu \(2020\)Hausdorff dimension, heavy tails, and generalization in neural networks\.Advances in Neural Information Processing Systems33,pp\. 5138–5151\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Sun, Y\. Yin, Z\. Xu, J\. Z\. Kolter, and Z\. Liu \(2025\)Idiosyncrasies in large language models\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Tang, X\. Li, F\. Liu, W\. Guo, H\. Zhou, Y\. Wang, K\. Han, X\. Yu, J\. Li, H\. Zang, F\. Mi, X\. Meng, Z\. Liu, H\. Chen, B\. Zheng, C\. Chen, Y\. Yan, R\. Tang, P\. Qin, X\. Chen, D\. Tao, and Y\. Wang \(2025\)Pangu pro moe: mixture of grouped experts for efficient sparsity\.External Links:2505\.21411,[Link](https://arxiv.org/abs/2505.21411)Cited by:[§5\.4](https://arxiv.org/html/2607.03377#S5.SS4.SSS0.Px1.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard,et al\.\(2023a\)LLaMA: open and efficient foundation language models\.External Links:2302\.13971,[Link](https://arxiv.org/abs/2302.13971)Cited by:[§5\.2](https://arxiv.org/html/2607.03377#S5.SS2.SSS0.Px1.p1.1)\.
- H\. Touvron, L\. Martin, K\. Stone,et al\.\(2023b\)Llama 2: open foundation and fine\-tuned chat models\.External Links:2307\.09288,[Link](https://arxiv.org/abs/2307.09288)Cited by:[§5\.1](https://arxiv.org/html/2607.03377#S5.SS1.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2607.03377#S5.SS2.SSS0.Px1.p1.1)\.
- Y\. Uchida, Y\. Nagai, S\. Sakazawa, and S\. Satoh \(2017\)Embedding watermarks into deep neural networks\.InProceedings of the 2017 ACM on International Conference on Multimedia Retrieval,ICMR ’17,New York, NY, USA,pp\. 269–277\.External Links:ISBN 9781450347013,[Link](https://doi.org/10.1145/3078971.3078974),[Document](https://dx.doi.org/10.1145/3078971.3078974)Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Wang, Z\. Ma, X\. Li, and Z\. Li \(2025\)Ghost in the transformer: tracing llm lineage with svd\-fingerprint\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.110.102.102.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.146.138.138.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.38.30.30.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.74.66.66.7)\.
- T\. Wang and F\. Kerschbaum \(2019\)Attacks on digital watermarks for deep neural networks\.In2019 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),Brighton, UK,pp\. 2622–2626\.External Links:[Document](https://dx.doi.org/10.1109/ICASSP.2019.8682202)Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Wu, L\. Pang, H\. Shen, X\. Cheng, and T\. Chua \(2023\)LLMDet: a third party large language models generated text detection tool\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 2113–2133\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.139/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.139)Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Wu, H\. Zhao, Z\. Wang, J\. Guo, Q\. Wang, and B\. He \(2025\)LLM dna: tracing model evolution via functional representations\.Cited by:[§5\.3](https://arxiv.org/html/2607.03377#S5.SS3.SSS0.Px2.p2.2),[§5\.3](https://arxiv.org/html/2607.03377#S5.SS3.SSS0.Px3.p1.1)\.
- Y\. Yang, L\. Hodgkinson, R\. Theisen, J\. Zou, J\. E\. Gonzalez, K\. Ramchandran, and M\. W\. Mahoney \(2021\)Taxonomizing local versus global structure in neural network loss landscapes\.InAdvances in Neural Information Processing Systems,Vol\.34,Virtual,pp\. 18722–18733\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Yang, R\. Theisen, L\. Hodgkinson, J\. E\. Gonzalez, K\. Ramchandran, C\. H\. Martin, and M\. W\. Mahoney \(2023\)Test accuracy vs\. generalization gap: model selection in nlp without accessing training or testing data\.InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,Long Beach, CA, USA,pp\. 3011–3021\.Cited by:[§1](https://arxiv.org/html/2607.03377#S1.p2.1),[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2607.03377#S3.SS0.SSS0.Px2.p2.5)\.
- Z\. Yang and H\. Wu \(2024\)A fingerprint for large language models\.Vol\.abs/2407\.01235\.External Links:[Link](https://api.semanticscholar.org/CorpusID:270869924)Cited by:[Table 1](https://arxiv.org/html/2607.03377#S1.T1.6.1.2.1),[§5\.1](https://arxiv.org/html/2607.03377#S5.SS1.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.134.126.126.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.26.18.18.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.62.54.54.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.98.90.90.7)\.
- D\. Yoon, M\. Chun, T\. Allen, H\. Müller, M\. Wang, and R\. Sharma \(2025\)Intrinsic fingerprint of llms: continue training is not all you need to steal a model\!\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 4791–4800\.External Links:[Link](https://aclanthology.org/P19-1472/),[Document](https://dx.doi.org/10.18653/v1/P19-1472)Cited by:[§5\.3](https://arxiv.org/html/2607.03377#S5.SS3.SSS0.Px2.p1.2)\.
- B\. Zeng, L\. Chen, Z\. He, X\. Wang, and Z\. Lin \(2025\)AWM: accurate weight\-matrix fingerprint for large language models\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2607.03377#S4.SS2.SSS0.Px1.p1.18),[§5\.1](https://arxiv.org/html/2607.03377#S5.SS1.SSS0.Px2.p1.1),[§5\.3](https://arxiv.org/html/2607.03377#S5.SS3.SSS0.Px2.p3.1),[§5\.5](https://arxiv.org/html/2607.03377#S5.SS5.p2.1)\.
- B\. Zeng, L\. Wang, Y\. Hu, Y\. Xu, C\. Zhou, X\. Wang, Y\. Yu, and Z\. Lin \(2024\)HuRef: HUman\-REadable fingerprint for large language models\.InAdvances in Neural Information Processing Systems,Vol\.37,Vancouver, Canada,pp\. 126332–126362\.Cited by:[Table 1](https://arxiv.org/html/2607.03377#S1.T1.6.1.4.1),[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2607.03377#S5.SS1.SSS0.Px2.p1.1),[§5\.3](https://arxiv.org/html/2607.03377#S5.SS3.SSS0.Px2.p3.1),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.128.120.120.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.20.12.12.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.56.48.48.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.92.84.84.7)\.
- J\. Zhang, D\. Liu, C\. Qian, L\. Zhang, Y\. Liu, Y\. Qiao, and J\. Shao \(2025\)REEF: representation encoding fingerprints for large language models\.InInternational Conference on Learning Representations,Singapore,pp\. 48092–48117\.Cited by:[Table 1](https://arxiv.org/html/2607.03377#S1.T1.6.1.3.1),[§5\.1](https://arxiv.org/html/2607.03377#S5.SS1.SSS0.Px2.p1.1),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.104.96.96.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.140.132.132.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.32.24.24.7),[Table 2](https://arxiv.org/html/2607.03377#S5.T2.68.60.60.7)\.
- Y\. Zhou, T\. Pang, K\. Liu, M\. W\. Mahoney, Y\. Yang,et al\.\(2023\)Temperature balancing, layer\-wise weight analysis, and neural network training\.Advances in Neural Information Processing Systems36,pp\. 63542–63572\.Cited by:[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2607.03377#S3.SS0.SSS0.Px2.p2.5)\.
- S\. Zhu, A\. M\. Ahmed, R\. Kuditipudi, and P\. Liang \(2025\)Independence tests for language models\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research,Vancouver, Canada,pp\. 79673–79698\.Cited by:[§1](https://arxiv.org/html/2607.03377#S1.p1.1),[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px1.p1.1),[§5\.4](https://arxiv.org/html/2607.03377#S5.SS4.SSS0.Px2.p1.2)\.
- R\. Zhuang, T\. Wu, Z\. Wen, A\. Li, J\. Jiao, and K\. Ramchandran \(2024\)EmbedLLM: learning compact representations of large language models\.External Links:2410\.02223,[Link](https://arxiv.org/abs/2410.02223)Cited by:[§1](https://arxiv.org/html/2607.03377#S1.p1.1),[§2](https://arxiv.org/html/2607.03377#S2.SS0.SSS0.Px2.p1.1),[§5\.3](https://arxiv.org/html/2607.03377#S5.SS3.SSS0.Px2.p2.2)\.

## 7\.Appendix

### 7\.1\.Robustness of ESD Shape Metrics

Pre\-training and post\-training reshape the correlations within weight matrices, as reflected in variations in the heavy\-tailed index across layers\. To validate the reliability of our method, we investigate whether the spectral structure preserves sufficient lineage information under post\-training perturbations\.

#### PL\_Alpha\_Hill is robust under perturbation\.

Fig\.[7](https://arxiv.org/html/2607.03377#S7.F7)shows that ESD shape metrics can encode lineage information beyond pretraining\. Using Spearman correlation between layer\-wise PL\_Alpha\_Hill profiles, models derived from the same 7B base model exhibit high similarities, whereas similarities across base families are substantially lower\. Moreover, within the Llama\-2 lineage, we observe finer sub\-structure: distinct clusters align with different post\-training trajectories \(e\.g\., instruction\-tuned vs\. CodeLlama\-style derivatives\), suggesting that post\-training alters spectral signatures in a structured manner\. These observations motivate our use of spectral signatures for downstream tasks such as lineage attribution, clustering, and performance prediction \(Sec\.[5](https://arxiv.org/html/2607.03377#S5)\)\.

![Refer to caption](https://arxiv.org/html/2607.03377v1/figs/signature/AverageSR.png)Figure 7\.Heatmap of spectral signature similarity across models derived from distinct 7B base models, measured by Spearman’s rank correlation of PL\_Alpha\_Hill\. Axis colors indicate base\-model lineage; dark and light green denote Llama\-2\-7B and CodeLlama\-7B\-hf derivatives, respectively\. The clear block structure shows that spectral signatures preserve ancestry despite diverse fine\-tuning trajectories\.
#### The Forging of Spectral Signature

As illustrated in Fig\.[8](https://arxiv.org/html/2607.03377#S7.F8), we observe three distinct phases of spectral evolution:

1. \(1\)Initialization\.At Step 0, all layers exhibit large, uniform PL\_Alpha\_Hill values \(metrics for attention and MLP weight matrices differ due to the aspect ratio difference\(Huet al\.,[2025](https://arxiv.org/html/2607.03377#bib.bib17)\)\), resembling MP distribution for random matrices\(Marčenko and Pastur,[1967](https://arxiv.org/html/2607.03377#bib.bib65)\)\.
2. \(2\)Formulation\.As training progresses, PL\_Alpha\_Hill decreases globally, signifying feature learning\. Crucially, PL\_Alpha\_Hill drops in early layers, flattens in middle layers, dips in mid\-deep layers, and rises in the final layers\.
3. \(3\)StabilizationAs shown in Fig\.[8](https://arxiv.org/html/2607.03377#S7.F8), the spectral signature stabilizes rapidly\. While the early phase \(<16​k<16ksteps\) shows high volatility, the signature “locks in” after step 16,000\. Subsequent updates refine the weights but do not alter the fundamental topological shape\.

#### PL\_Alpha\_Hill undergo minimal change during post\-training\.

Because post\-training updates \(e\.g\., SFT, RLVR\) introduce relatively small changes in parameter space, they do not erase the heavy\-tailed structure established during pre\-training\. Based on that, PL\_Alpha\_Hill works as a stable and lineage\-preserving signature that remains robust to common downstream adaptations\. This motivates us to use it for performance prediction and model attribution\.

![Refer to caption](https://arxiv.org/html/2607.03377v1/figs/Pythia/layerwise_alpha_evolution.png)Figure 8\.The evolution of layer\-wise PL\_Alpha\_Hill through the training process\. The values of PL\_Alpha\_Hill are averaged within Attention modules and MLP modules\.

### 7\.2\.Metrics for Classification Quality

Given two signatures𝐙1,𝐙2\\mathbf\{Z\}\_\{1\},\\mathbf\{Z\}\_\{2\}, we consider: \(i\) Spearman rank correlationρs​\(𝐙1,𝐙2\)\\rho\_\{s\}\(\\mathbf\{Z\}\_\{1\},\\mathbf\{Z\}\_\{2\}\)as our default similarity \(Eq\.[3](https://arxiv.org/html/2607.03377#S4.E3)\), \(ii\) an RBF kernel applied to the signature distance,

\(6\)kRBF​\(𝐙1,𝐙2\)=exp⁡\(−‖𝐙1−𝐙2‖222​σ2\)k\_\{\\text\{RBF\}\}\(\\mathbf\{Z\}\_\{1\},\\mathbf\{Z\}\_\{2\}\)=\\exp\\\!\\left\(\-\\frac\{\\\|\\mathbf\{Z\}\_\{1\}\-\\mathbf\{Z\}\_\{2\}\\\|\_\{2\}^\{2\}\}\{2\\sigma^\{2\}\}\\right\), and \(iii\) cosine similarity,

\(7\)kcos​\(Z1,𝐙2\)=⟨𝐙1,𝐙2⟩‖𝐙1‖2​‖𝐙2‖2\.k\_\{\\cos\}\\mathbf\{\(\}\{Z\}\_\{1\},\\mathbf\{Z\}\_\{2\}\)=\\frac\{\\langle\\mathbf\{Z\}\_\{1\},\\mathbf\{Z\}\_\{2\}\\rangle\}\{\\\|\\mathbf\{Z\}\_\{1\}\\\|\_\{2\}\\\|\\mathbf\{Z\}\_\{2\}\\\|\_\{2\}\}\.

### 7\.3\.Metrics for Internal Clustering Validation

For unsupervised model\-family discovery, we evaluate spectral\-signature clusters using two standard internal validity indices: the Silhouette Coefficient and the Davies\-Bouldin Index\.

#### Silhouette Coefficient\.

The Silhouette Coefficient measures clustering quality by comparing intra\-cluster cohesion and inter\-cluster separation\. For a pointi∈Cki\\in C\_\{k\}, leta​\(i\)a\(i\)denote its average distance to other points inCkC\_\{k\}, and letb​\(i\)b\(i\)denote the smallest average distance to points in any other cluster\. The silhouette coefficient is

\(8\)s​\(i\)=b​\(i\)−a​\(i\)max⁡a​\(i\),b​\(i\)\.s\(i\)=\\frac\{b\(i\)\-a\(i\)\}\{\\max\{a\(i\),b\(i\)\}\}\.The overall Silhouette Score is the average ofs​\(i\)s\(i\)over all samples\.

#### Davies\-Bouldin Index\.

The Davies\-Bouldin Index \(DBI\) compares intra\-cluster dispersion with inter\-cluster separation\. LetSiS\_\{i\}denote the dispersion of clusterii, andMi​jM\_\{ij\}denote the distance between the centroids of clustersiiandjj\. Define

\(9\)Ri​j=Si\+SjMi​j\.R\_\{ij\}=\\frac\{S\_\{i\}\+S\_\{j\}\}\{M\_\{ij\}\}\.Then

\(10\)D​B=1k​∑i=1kmaxj≠i⁡Ri​j\.DB=\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\max\_\{j\\neq i\}R\_\{ij\}\.

### 7\.4\.Additional Results on Model Clustering

In Fig\.[9](https://arxiv.org/html/2607.03377#S7.F9), we show a t\-SNE visualization of a 2D embedding for spectral clustering on PL\_Alpha\_Hill with ground truth labels\.

![Refer to caption](https://arxiv.org/html/2607.03377v1/x7.png)

![Refer to caption](https://arxiv.org/html/2607.03377v1/x8.png)

Figure 9\.t\-SNE 2D embedding for spectral clustering on PL\_Alpha\_Hill with ground truth cluster labels\.
### 7\.5\.Additional Generalization Tests

#### Model Scale Generalization\.

As shown in Tab\.[10](https://arxiv.org/html/2607.03377#S7.T10), we evaluate additional models at 1\.3B \(GPT\-Neo family\) and 70B \(Llama\-3\.1 family\); related pairs achieve similarity≥\\geq0\.99 while unrelated pairs score 0\.55–0\.69, consistent with the main results\.

Table 10\.Similarity scores for additional model scales\.ModelRelationSimilarityLlama\-3\.1\-70B\-InstructRelated0\.99OpenMath2\-Llama3\.1\-70BRelated0\.99Qwen2\.5\-72BUnrelated0\.55lamini\-neo\-1\.3BRelated1\.00lamini\-cerebras\-1\.3BUnrelated0\.68Quokka\-1\.3BUnrelated0\.69

#### Intra\-family Performance Prediction\.

To further validate that spectral signatures capture within\-family performance variation, we evaluate fine\-tuned models with substantial benchmark gains over their base models \(\>5\>5points\)\. As shown in Tab\.[11](https://arxiv.org/html/2607.03377#S7.T11), the predicted scores track the true scores rather than collapsing to the base\-model prediction\. For the Llama\-13B family, our method achieves MAEs of 1\.94 on TruthfulQA and 1\.55 on MMLU, with significant rank correlation in 7 out of 8 settings\.

Table 11\.Intra\-family performance prediction using statistically significant Kendall’s Tau correlation\.BenchmarkLlama\-2\-13bLlama\-13bMMLU0\.83470\.83470\.00040\.0004ARC\-C0\.02720\.02720\.00020\.0002HellaSwag0\.00610\.00610\.00480\.0048TruthfulQA0\.00860\.00860\.00910\.0091

### 7\.6\.Additional Details on Ablation Studies

#### Output\-Invariant Reparameterization

We apply scale\-preserving reparameterization to consecutive linear maps while keeping the composed operator unchanged\. We define a scaling factora=exp⁡\(γ​z\)a=\\exp\(\\gamma z\)with strengthγ\\gammaandz∼𝒩​\(0,1\)z\\sim\\mathcal\{N\}\(0,1\)\. The weights are updated asQ′=a​Q,K′=K/aQ^\{\\prime\}=aQ,K^\{\\prime\}=K/a\(and similarly forV,OV,O\)\. For Permutation on Hidden Units, we generate a random permutation matrixPPand construct a transformed weight set:W′=\{P​Wg​a​t​e,P​Wu​p,Wd​o​w​n​P−1\}W^\{\\prime\}=\\\{PW\_\{gate\},PW\_\{up\},W\_\{down\}P^\{\-1\}\\\}for each layer\. The similarity between reparameterized models and their origins remains 1 for Llama\-2\-7b, Qwen\-1\.5\-7b, and Mistral\-7b, suggesting robustness of our method\. In contrast, PCS degrades substantially under both transformations\.

#### Noise Perturbation

For each weight matrixWℓW\_\{\\ell\}, we sampleZℓ∼𝒩​\(0,I\)Z\_\{\\ell\}\\sim\\mathcal\{N\}\(0,I\)with the same shape and define

\(11\)Δ​Wℓ=std​\(Wℓ\)​Zℓ,Wℓ\(γ\)=Wℓ\+γ​Δ​Wℓ,\\Delta W\_\{\\ell\}=\\mathrm\{std\}\(W\_\{\\ell\}\)Z\_\{\\ell\},\\qquad W\_\{\\ell\}^\{\(\\gamma\)\}=W\_\{\\ell\}\+\\gamma\\Delta W\_\{\\ell\},whereγ\\gammacontrols the noise strength\. We pre\-generateΔ​Wℓℓ\{\\Delta W\_\{\\ell\}\}\_\{\\ell\}once and reuse it across allγ\\gammavalues for controlled comparisons\.