Reference-Based Distillation Detection in LLMs

# Reference-Based Distillation Detection in LLMs
Source: [https://arxiv.org/html/2607.09692](https://arxiv.org/html/2607.09692)
Rajat Rawat1Sizhe Chen1Akshay Anand1Michael Duan2 Bob Rotsted3Sewon Min1 1University of California, Berkeley2University of Southern California3OpenAI rajat\_rawat@berkeley\.edu

###### Abstract

Model distillation—training on outputs from stronger third\-party models—is widely used to boost performance, but raises concerns about unfair advantages and policy violations\. This motivates a fundamental question:*can we detect whether a model was distilled from another?*We show that, while identifying a teacher model from a student in isolation is highly challenging, it becomes tractable in areference\-based setting: given a model and an earlier\-generation checkpoint from the same lineage, we can identify the teacher model used to train the later checkpoint\. We introduce a distillation detection method based on reference\-based membership inference\. By comparing how strongly a student model preferentially aligns with outputs from different candidate teachers relative to a reference checkpoint, our method identifies the most likely teacher and detects evidence of distillation\. To handle unknown distillation pipelines such as hidden prompts, we infer proxy prompt templates directly from model outputs\. We additionally identify a distinctive glyph\-level signal specific to o1/o3 models\. Evaluating distillation detection is challenging because modern model lineages are already heavily entangled\. To address this, we develop a hybrid evaluation spanning both controlled distillation experiments and real\-world models\. Across both settings, our approach recovers the true teacher with near\-perfect accuracy in single\-teacher distillation scenarios, even when the underlying distillation pipeline is largely unknown\. We further introduce statistical tests for both teacher attribution and distillation detection, and extend our framework to open\-world settings where no teacher is guaranteed to be present among the candidates\. Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek\-R1, and GPT\-OSS\.111Code is available at[https://github\.com/RajatRawat\-creator/DistillDetect](https://github.com/RajatRawat-creator/DistillDetect)\.

## 1Introduction

Large language models \(LLMs\) are increasingly trained via model distillation, where a student is fine\-tuned on outputs from a stronger teacher\. This approach has become a standard way to transfer capabilities such as reasoning and instruction following\[[7](https://arxiv.org/html/2607.09692#bib.bib1),[27](https://arxiv.org/html/2607.09692#bib.bib2),[26](https://arxiv.org/html/2607.09692#bib.bib4),[13](https://arxiv.org/html/2607.09692#bib.bib5)\]\. While effective and widely adopted, distillation creates ecosystem\-level risks: a competitor can query a proprietary model at scale and train a competing system on its outputs, cheaply acquiring capabilities that would otherwise require new architectures or datasets\. This raises concerns about unfair advantage and policy violations\. Major providers \(e\.g\., Google, OpenAI, Anthropic\) have responded with system\-level defenses based on request metadata\[[14](https://arxiv.org/html/2607.09692#bib.bib15),[22](https://arxiv.org/html/2607.09692#bib.bib17),[3](https://arxiv.org/html/2607.09692#bib.bib16)\], but these require usage logs and infrastructure access and do not address detection from model behavior\.

This motivates a fundamental problem in model auditing:*given a model, can we determine whether a teacher was used to train it?*Answering this is important for intellectual property protection, accountability, and transparency when training details are undisclosed, but is challenging due to both the difficulty of membership inference in modern LLMs and the opacity of distillation pipelines, including unknown prompts, data sources, and filtering\. Detecting distillation in this absolute sense is quite difficult; however, we find that a reference\-based setting is more tractable:*given a model and an earlier\-generation checkpoint of itself, can we determine whether a teacher was used to train the later checkpoint from the earlier one, and if so, which teacher it was?*

![Refer to caption](https://arxiv.org/html/2607.09692v1/x1.png)Figure 1:Reference\-based distillation detection\. \(a\) Rather than attributing distillation from the final student alone, we measure the shift relative to a reference model \(e\.g\., an earlier model from the same family\)\. \(b\) Scoring candidate teachers for QwQ\-32B identifies DeepSeek R1 \(red\) as the best match\.In this work, we propose a teacher identification method based on contrastive, reference\-normalized likelihoods \(§[3](https://arxiv.org/html/2607.09692#S3); Figure[1](https://arxiv.org/html/2607.09692#S1.F1)\(a\)\)\. Given candidate teachers, we score each by how strongly the student prefers that teacher’s outputs relative to a reference model, and select the teacher whose signal consistently dominates\. The key insight is that likelihoods taken relative to the earlier checkpoint—not absolute ones—carry the signal: standard membership inference metrics \(e\.g\., raw likelihood, ZLIB\-normalized likelihood, Min\-KKprobability\) fail to reliably recover the true teacher, whereas our reference\-based formulation remains robust even under opaque and heterogeneous pipelines\. The method requires only black\-box access, extends naturally to hidden\-prompt settings via proxy prompt inference, and is paired with a per\-probe significance test for teacher attribution\.

Beyond likelihood\-based attribution, we identify a distinctive glyph\-level signature of o1/o3 models: unusually frequent use of non\-ASCII Unicode characters\. We exploit this signal through a reference\-normalized loss gap between ASCII and Unicode serializations, yielding a complementary indicator of o1/o3 distillation that persists in distilled students\.

Evaluating distillation detection is itself challenging because model lineage is already deeply entangled across modern LLMs\. We therefore design a hybrid evaluation method combining controlled experiments and real\-world models \(§[4](https://arxiv.org/html/2607.09692#S4)\), focusing on a canonical setting: single\-teacher, last\-stage, SFT\-style distillation\. Controlled experiments use student–teacher pairs known not to be distilled from one another, ensuring clean ground truth at the cost of a more synthetic setup; real\-world models better reflect practical conditions but introduce ambiguity, as candidate teachers may themselves be distilled from each other\. Across both settings, our method reliably recovers the true teacher with near\-perfect accuracy, significantly outperforming baselines, and remains robust under domain shift between distillation and detection data\. To separate genuine distillation from coincidental similarity, we pair the detection threshold with a significance test on the per\-probe margin and require agreement across two separate prompt sets\. Cross\-validation results suggest that the resulting threshold generalizes well to previously unseen student models\.

Finally, we extend our framework to the open\-world setting, where no teacher is guaranteed to exist among the candidates tested, and apply it to contemporary models to examine open provenance questions \(§[7](https://arxiv.org/html/2607.09692#S7); Figure[1](https://arxiv.org/html/2607.09692#S1.F1)\(b\)\)\. Through case studies on QwQ, DeepSeek\-R1, and GPT\-OSS, we quantify empirical evidence of potential distillation and identify likely teacher sources when such signals exist\.

Together, these findings establish the feasibility of model\-level auditing for distillation in modern LLMs, advancing transparency and accountability in increasingly opaque model development pipelines\. While our study focuses on a simplified setting, we hope this work motivates future research on attribution under more complex regimes, including multi\-teacher and multi\-stage distillation\.

## 2Related Work

#### Distillation in LLMs

Model distillation was introduced for compressing knowledge into smaller models in image models\[[7](https://arxiv.org/html/2607.09692#bib.bib1)\]\. Recent work has adopted distillation to obtain strong LLMs\[[9](https://arxiv.org/html/2607.09692#bib.bib9),[21](https://arxiv.org/html/2607.09692#bib.bib10),[8](https://arxiv.org/html/2607.09692#bib.bib11)\], such as Phi\[[1](https://arxiv.org/html/2607.09692#bib.bib18)\], Alpaca\[[25](https://arxiv.org/html/2607.09692#bib.bib19)\], etc\. The performance gain from distillation is improved when the teacher is a reasoning LLM\[[13](https://arxiv.org/html/2607.09692#bib.bib5)\]\.

While effective and widely adopted, distillation creates ecosystem\-level risks: model developers may query competitor models at scale and train systems on their outputs to replicate capabilities at a fraction of the original development cost\. This has emerged as an increasingly imminent concern, with industry\-scale distillation repeatedly reported by major AI providers\[[14](https://arxiv.org/html/2607.09692#bib.bib15),[3](https://arxiv.org/html/2607.09692#bib.bib16),[22](https://arxiv.org/html/2607.09692#bib.bib17)\]\.

#### Distillation detection in LLMs

The growing use of distillation to reproduce competitor capabilities has consequently prompted major AI providers to deploy detection systems\. Existing industry detection primarily relies on request\-side monitoring and infrastructure signals rather than model behavior itself, such as request metadata, IP address correlation, and infrastructure\-level indicators\[[3](https://arxiv.org/html/2607.09692#bib.bib16)\]\.

Another line of work studies watermark\-based detection, where the auditor controls the teacher model and embeds detectable signals into generated outputs\[[20](https://arxiv.org/html/2607.09692#bib.bib28)\]\. However, these approaches require strong assumptions about provider\-side access, and recent work shows that such watermarks can often be weakened or removed during downstream training\[[16](https://arxiv.org/html/2607.09692#bib.bib29)\]\.

\[[30](https://arxiv.org/html/2607.09692#bib.bib32)\]studies a different but related problem: detecting which prompts are used for generating continuations used in a model’s distillation\.

In summary, distillation detection has attracted growing attention in both industry and academia\. These approaches target absolute, provider\-level detection\. To the best of our knowledge, there is no prior work on reference\-based LLM detection from model behavior alone, and we present this work as a first step in that direction\.

#### Distillation detection in non\-LLM settings

Distillation detection has also been studied outside the LLM setting, particularly for image models\. Prior work detects distilled image classifiers and generative models using synthesized inputs and statistical scoring\[[23](https://arxiv.org/html/2607.09692#bib.bib13)\], prompt\-level fingerprints in text\-to\-image models\[[31](https://arxiv.org/html/2607.09692#bib.bib31)\], and shadow\-model\-based provenance verification comparing generalization behavior against suspicious models\[[29](https://arxiv.org/html/2607.09692#bib.bib30)\]\.

#### Membership Inference

Our approach relies on membership inference, which determines whether a particular example was used during model training\[[24](https://arxiv.org/html/2607.09692#bib.bib6)\]according to the intuition that training examples receive lower loss than non\-training examples\. It has been shown that comparing against a reference output distribution yields stronger membership inference signal\[[5](https://arxiv.org/html/2607.09692#bib.bib7),[28](https://arxiv.org/html/2607.09692#bib.bib33)\]\. A recent work extends membership inference to decide whether a model is trained on a dataset\[[10](https://arxiv.org/html/2607.09692#bib.bib14)\]\. We build on these literature but study a different problem: whether a model is trained on another model’s generations\.

## 3Method

#### LLM Distillation

We consider supervised fine\-tuning \(SFT\)\-based distillation, a common approach for transferring behavior from a teacher LLM to a smaller student\. LetS0S\_\{0\}be a base student model andTTa teacher model\. Given inputs𝒳=\{xi\}i=1N\\mathcal\{X\}=\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}, where eachxix\_\{i\}includes the prompt template used to query the teacher, the teacher produces outputsyi=T​\(xi\)y\_\{i\}=T\(x\_\{i\}\), yielding the distillation dataset𝒟T=\{\(xi,yi\)\}i=1N\\mathcal\{D\}\_\{T\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}\. Fine\-tuningS0S\_\{0\}on𝒟T\\mathcal\{D\}\_\{T\}produces the distilled studentSS\.

#### Distillation detection

Given a subject modelSSand candidate teachers𝒯=\{T1,…,TK\}\\mathcal\{T\}=\\\{T\_\{1\},\\dots,T\_\{K\}\\\}, our goal is to determine whetherSSwas distilled from any teacher in𝒯\\mathcal\{T\}, and if so, identify the teacher\. As an initial step, we assumeSSwas distilled from at most one teacher in𝒯\\mathcal\{T\}\. We assume the detector has logits access toSSand𝒯\\mathcal\{T\}, but not to the prompts or templates used for distillation; instead, the detector queriesSSusing its default chat template and public prompts, testing whether distillation signals remain detectable under this limited\-information setting\.

A natural first attempt is to detect distillation in an*absolute*sense: scoring candidate teacher outputs directly underSS—for instance, via raw likelihood or standard membership\-inference metrics—without any point of comparison\. We find this unreliable\. We instead adopt a*reference\-based*formulation: rather than asking whetherSSexplains a teacher’s outputs in absolute terms, we ask whether it explains them*better than*an earlier\-generation reference checkpointRRdoes\. The rest of this section develops this reference\-based approach for teacher identification \(§[3\.1](https://arxiv.org/html/2607.09692#S3.SS1)\) and the binary detection problem \(§[3\.2](https://arxiv.org/html/2607.09692#S3.SS2)\)\.

### 3\.1Teacher Identification

We first consider the simpler setting ofteacher identification, assuming a single true teacher exists in𝒯\\mathcal\{T\}, as also studied in prior computer vision work\[[23](https://arxiv.org/html/2607.09692#bib.bib13)\]\. Formally, the task maps a studentSSand candidate set\{T1,…,TK\}\\\{T\_\{1\},\\ldots,T\_\{K\}\\\}to the index of the true teacher:\(S,\{T1,…,TK\}\)↦y∈\[K\]\.\(S,\\\{T\_\{1\},\\ldots,T\_\{K\}\\\}\)\\mapsto y\\in\[K\]\.Our approach proceeds as follows:

1. 1\.Construct proxy inputs𝒳^\\hat\{\\mathcal\{X\}\}from open\-source data, which may differ from the original inputs𝒳\\mathcal\{X\}\.
2. 2\.Query each teacherTkT\_\{k\}to obtain outputs for everyxi∈𝒳^x\_\{i\}\\in\\hat\{\\mathcal\{X\}\}\.
3. 3\.Use a scoring function𝐟\\mathbf\{f\}to compute alignment𝐟​\(Tk​\(xi\),S\)\\mathbf\{f\}\(T\_\{k\}\(x\_\{i\}\),S\)between each teacher output andSS\.

We identify the teacher using two complementary decision rules: \(i\) selecting the teacher with the highest mean alignment score across𝒳^\\hat\{\\mathcal\{X\}\}, and \(ii\) sorting per\-teacher scores over𝒳^\\hat\{\\mathcal\{X\}\}and selecting the teacher that wins the largest fraction of rank\-matched comparisons\. The latter captures distributional preference without requiring per\-prompt dominance\. We reportPer\-sampleas the fraction of rank\-matched comparisons won by the true teacher, andAgg\.as the better of the two decision rules\.

#### How to choose𝐟\\mathbf\{f\}?

Concretely, the non reference\-based approach instantiates𝐟\\mathbf\{f\}as a direct comparison betweenS​\(x\)S\(x\)andTk​\(x\)T\_\{k\}\(x\)—via lexical or semantic similarity \(nn\-gram overlap, embedding metrics\), or by scoringTk​\(x\)T\_\{k\}\(x\)by its log\-likelihood underSS\. We find these unreliable: log\-likelihood, in particular, can favor teacher outputs that are easy to model in general rather than specifically aligned withSS\(empirical evidence in Table[1](https://arxiv.org/html/2607.09692#S4.T1)\)\.

This motivates the reference\-based formulation\. Following reference\-based membership inference\[[28](https://arxiv.org/html/2607.09692#bib.bib33)\], we calibrate the log\-likelihood underSSusing an earlier generation checkpoint reference modelRR, which was ideally not exposed to any teacher candidate, or otherwise less affected by the distillation\. For example,RRmay be an earlier checkpoint in the same family \(e\.g\., Llama 3\.1 Instruct asRRfor Llama 3\.2 Instruct\), a prior training stage \(e\.g\., a base model whenSSis instruction\-tuned\), or a smaller variant \(e\.g\., GPT\-OSS 20B asRRfor GPT\-OSS 120B\)\. We evaluate several choices ofRR\(§[B\.1](https://arxiv.org/html/2607.09692#A2.SS1)\) and find that a suitable reference can typically be identified in realistic auditing scenarios\.

For each example, we compute the normalized alignment score

𝐟​\(Tk​\(xi\),S\)=−\(ℓS​\(xi,yi\(k\)\)−ℓR​\(xi,yi\(k\)\)\),\\mathbf\{f\}\(T\_\{k\}\(x\_\{i\}\),S\)=\-\\big\(\\ell\_\{S\}\(x\_\{i\},y\_\{i\}^\{\(k\)\}\)\-\\ell\_\{R\}\(x\_\{i\},y\_\{i\}^\{\(k\)\}\)\\big\),whereℓM​\(x,y\)\\ell\_\{M\}\(x,y\)is the average token\-level negative log\-likelihood ofyygivenxxunder modelMM\. This measures how much better the student explains a candidate teacher’s output relative to the reference\. We aggregate these scores per teacher using the mean\-score and sorted per\-sample rules above\.

### 3\.2Distillation Detection

Beyond identifying a single teacher, we consider the binary*distillation detection*problem: givenSSand candidate set\{T1,…,TK\}\\\{T\_\{1\},\\ldots,T\_\{K\}\\\}, predicty∈\{0,1\}y\\in\\\{0,1\\\}indicating whetherSSwas distilled from any teacher in the set\.222We assumeSSis distilled from at most one candidate teacher\.Using the reference\-normalized scores𝐟\\mathbf\{f\}from §[3\.1](https://arxiv.org/html/2607.09692#S3.SS1), we define themarginδ=s¯1−s¯2\\delta=\\bar\{s\}\_\{1\}\-\\bar\{s\}\_\{2\}, wheres¯k=1N​∑i=1N𝐟​\(Tk​\(xi\),S\)\\bar\{s\}\_\{k\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{f\}\(T\_\{k\}\(x\_\{i\}\),S\)ands¯1≥s¯2\\bar\{s\}\_\{1\}\\geq\\bar\{s\}\_\{2\}are the top two mean scores among candidates\. If the true teacher is present, it should dominate the pool, yielding a largeδ\\delta; if absent, scores are more uniform andδ\\deltastays small\. Given a calibration set, we select a thresholdτ\\tauby trying each observed training margin as a cutpoint and choosing the value that maximizes F1 on the calibration examples\. Crucially,τ\\tauis selected using only the training examples and never the held\-out margins\. For each held\-out evaluation cell, its margin is compared against the corresponding fold\-specific threshold\. The final model\-level detection additionally requires the margin to significantly exceedτ\\tauon both prompt sets and for both prompt sets to identify the same teacher, as defined in §[3\.3](https://arxiv.org/html/2607.09692#S3.SS3)\.

### 3\.3Statistical Significance

Both tasks above output a point estimate—a teacher ranking for identification and a marginδ\\deltafor detection—so for each we add a test that checks whether the signal is real rather than noise\. We start with identification, where the true teacher is known\.

#### Per\-probe top\-1 consistency \(identification\)\.

With the true teachert⋆t^\{\\star\}known in advance, we ask the most direct question: on how many*individual*probes ist⋆t^\{\\star\}the top\-scoring candidate? We count a win on probexix\_\{i\}whent⋆t^\{\\star\}outscores every other candidate,

Xi=𝟙​\[arg⁡maxk⁡𝐟​\(Tk​\(xi\),S\)=t⋆\],X\_\{i\}=\\mathbbm\{1\}\\\!\\left\[\\arg\\max\_\{k\}\\,\\mathbf\{f\}\(T\_\{k\}\(x\_\{i\}\),S\)=t^\{\\star\}\\right\],and report the win rateρ^=1N​∑i=1NXi\\hat\{\\rho\}=\\tfrac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}X\_\{i\}; a*majority*meansρ^\>0\.5\\hat\{\\rho\}\>0\.5\. To rule out chance, note that if ranking were randomt⋆t^\{\\star\}would top a probe with probability1/K1/K, making theNNwins Binomial; we testH0:ρ≤1/KH\_\{0\}:\\rho\\leq 1/KagainstH1:ρ\>1/KH\_\{1\}:\\rho\>1/Kwith a one\-sided exact binomial test\. Because the test is exact, it needs no normality check or Wilcoxon fallback, and becauset⋆t^\{\\star\}is fixed in advance \(not picked as the best ofKK\), it needs no Bonferroni correction—which is why this is an identification test, complementing the detection test below\.

#### Margin significance \(detection\)\.

The marginδ\\deltais a point estimate; to decide whether an observedδ\\deltareflects distillation rather than sampling noise, we attach an inferential test\. Lett1t\_\{1\}andt2t\_\{2\}denote the top\- and second\-ranked candidates by mean score, and define the paired per\-probe difference

di=𝐟​\(Tt1​\(xi\),S\)−𝐟​\(Tt2​\(xi\),S\)\.d\_\{i\}=\\mathbf\{f\}\(T\_\{t\_\{1\}\}\(x\_\{i\}\),S\)\-\\mathbf\{f\}\(T\_\{t\_\{2\}\}\(x\_\{i\}\),S\)\.Pairing within a probe cancels per\-prompt difficulty shared across candidates, and the sample mean ofddrecoversδ\\delta\. We test against the calibrated thresholdτ∗\\tau^\{\*\}rather than against zero: atN≈200N\\\!\\approx\\\!200, even a negligible margin can be significant against zero, including for non\-distilled models\. Concretely, forH0:δ≤τ∗H\_\{0\}\\\!:\\delta\\leq\\tau^\{\*\}againstH1:δ\>τ∗H\_\{1\}\\\!:\\delta\>\\tau^\{\*\}, we use a one\-sidedtt\-test with statistict=\(δ−τ∗\)/\(sd/N\)t=\(\\delta\-\\tau^\{\*\}\)/\(s\_\{d\}/\\sqrt\{N\}\)onN−1N\\\!\-\\\!1degrees of freedom, and report a95%95\\%confidence interval forδ\\deltafrom10,00010\{,\}000probe\-level bootstrap resamples as an effect\-size diagnostic where shown \(Table[8](https://arxiv.org/html/2607.09692#S6.T8)\)\. Sinceddis occasionally non\-normal \(Shapiro–Wilk\), we additionally compute a one\-sided Wilcoxon signed\-rank test on the shifted differencesdi−τ∗d\_\{i\}\-\\tau^\{\*\}—a distribution\-free check of the same hypothesis—and base the decision on this test whenever the normality assumption is rejected \(Shapirop<0\.05p<0\.05\)\. The decision uses the margin over the runner\-upt2t\_\{2\}: sincet2t\_\{2\}is the closest competitor, clearingτ∗\\tau^\{\*\}against it would generally imply clearingτ∗\\tau^\{\*\}against every weaker candidateTkT\_\{k\}\(k≠t1k\\neq t\_\{1\}\)\.

Becauset1t\_\{1\}is selected as the best ofKKcandidates, we Bonferroni\-correct the selection by reportingp∗=min⁡\(1,p⋅\(K−1\)\)p^\{\*\}=\\min\(1,\\,p\\cdot\(K\\\!\-\\\!1\)\), whereppis thepp\-value of the governing test \(ttor Wilcoxon\)\. We call a target distilled from teacherTTonly if the corrected test rejectsH0H\_\{0\}\(p∗<0\.05p^\{\*\}<0\.05\) on*both*prompt sets and both name the same teacher; requiring cross\-prompt agreement makes the call teacher\-specific rather than family\-generic\.

## 4Experiment: Teacher Identification

We first validate our teacher identification method\.

#### Disclaimer\.

Our evaluation focuses on a canonical setting—single\-teacher, SFT\-style final\-stage distillation—and isolates this minimal setup; real\-world pipelines may involve multiple teachers, multi\-stage use, and diverse roles \(e\.g\., generation, rewriting\), which we defer to future work\.

### 4\.1Experimental Setup

Because modern LLM lineages are deeply entangled, ground\-truth attribution is often ambiguous\. We therefore use a hybrid evaluation:controlled experiments, where we distill models ourselves to isolate key variables, andmodels in the wild, where real\-world distilled models test robustness under unknown prompts, instructions, filtering, and training pipelines\.

#### Controlled Experiments

Since most state\-of\-the\-art LLMs may already have been distilled from each other, we select teacher–student pairs from distinct U\.S\.\-organization model families, where legal constraints limit cross\-company distillation\[[17](https://arxiv.org/html/2607.09692#bib.bib20)\], and additionally include Qwen models to broaden coverage—though we acknowledge that prior distillation is harder to rule out in these cases\.

We consider four base student models \(S0S\_\{0\}\): Llama\-3\.2\-3B\-Instruct\[[12](https://arxiv.org/html/2607.09692#bib.bib21)\], Gemma\-3\-4B\-PT\[[6](https://arxiv.org/html/2607.09692#bib.bib22)\], Qwen2\.5\-1\.5B, and Qwen2\.5\-3B\[[18](https://arxiv.org/html/2607.09692#bib.bib23)\]\. We distill each from three teachers: Qwen3\-8B\[[19](https://arxiv.org/html/2607.09692#bib.bib25)\], Llama\-3\.3\-70B\-Instruct\[[11](https://arxiv.org/html/2607.09692#bib.bib24)\], and GPT\-OSS\-120B\[[15](https://arxiv.org/html/2607.09692#bib.bib26)\], using prompts from OpenMathInstruct\-2 \(OMI\-2\) and s1\[[26](https://arxiv.org/html/2607.09692#bib.bib4),[13](https://arxiv.org/html/2607.09692#bib.bib5)\], yielding4×3×2=244\\times 3\\times 2=24distilled students\. We retain the 19 for which distillation improves utility over the base model\. At detection time, we additionally include Gemma\-3\-27B\-IT\[[6](https://arxiv.org/html/2607.09692#bib.bib22)\]as a candidate teacher, forming a four\-way classification task\.

#### Real\-World Distilled Models

For the pool of teacher candidates, we consider 10 models: Claude Opus 4\.5, Claude Opus 4\.6, Claude 3\.5 Sonnet, DeepSeek R1, GPT\-OSS\-120B, Gemma\-3\-27B\-it, Llama\-3\.3\-70B\-Instruct, QwQ\-32B\-Preview, o1, and o3—forming a ten\-way classification task\.

Table 1:Teacher identification accuracy \(%\) in controlled experiments\. Accuracy is broken down by true teacher model, including both aggregated and per\-sample metrics\. Our method, which is the reference\-based variant of likelihood, achieves near\-perfect accuracy\.Table 2:Per\-probe top\-1 significance in controlled experiments\.For each retained student, we count how often the true teacher is the top\-scoring candidate across the cross\-dataset probes \(s1→\\toOMI and OMI→\\tos1\) and test this win rate against chance with a one\-sided exact binomial test\. We report the range ofpp\-values across students for each teacher\.nnvaries because we exclude distilled students that underperform their base model\. All 19 retained students are significant\.
#### Detection method details\.

We use the pre\-distilled base model corresponding to the studentSSas a reference model \(RR\), e\.g\., ifSSis obtained by distilling a base model, we use that original base model asRR\. We perform ablations on this choice\. We also truncate all teacher outputs to 2,048 tokens to prevent output\-length variation from trivializing detection\.

As proxy prompt datasets𝒳^\\hat\{\\mathcal\{X\}\}: we assume no access to the original distillation inputs𝒳\\mathcal\{X\}and use disjoint inputs, e\.g\., s1 for models distilled on OMI prompts, and vice versa\. For Distilled R1 models, since the true distillation prompts are truly unknown, we use both s1 and OMI as proxy prompts \(𝒳^\\hat\{\\mathcal\{X\}\}\)\.

We compare against length\-matching, string\-based \(multinn\-gram Jaccard, symmetric coverage, embedding cosine\), and membership\-inference baselines \(raw likelihood, ZLIB\-normalized, Min\-kk, and reverse variants\)\. §[A](https://arxiv.org/html/2607.09692#A1)provides the complete controlled\-experiment configurations, distillation hyperparameters, and baseline definitions\. We report both sorted per\-sample \(Per\-sample\) and aggregated \(Agg\.\) accuracy, as defined in §[3\.1](https://arxiv.org/html/2607.09692#S3.SS1)\.

### 4\.2Results from Controlled Experiments

Table[1](https://arxiv.org/html/2607.09692#S4.T1)reports results across the 19 retained distilled students\. Our reference\-based method achieves near\-perfect accuracy in every setting, while baselines are inconsistent—performing well only when Llama 3\.3 70B Instruct is the teacher and failing on Qwen\-3\-8B and GPT\-OSS\-120B\.

![Refer to caption](https://arxiv.org/html/2607.09692v1/x2.png)Figure 2:Reference\-based MIA CDFs\.Qwen\-2\.5\-3B distilled from GPT\-OSS\-120B on s1, probed with OMI\.Figure[2](https://arxiv.org/html/2607.09692#S4.F2)shows a representative controlled example: a Qwen\-2\.5\-3B\-Base student distilled from GPT\-OSS\-120B on s1 prompts\. The true teacher is clearly separated under the non\-training data probes \(OMI\), and we observe the same qualitative pattern across all controlled student–teacher pairs\.

Statistical significance testing in Table[2](https://arxiv.org/html/2607.09692#S4.T2)further confirms that the true teacher’s per\-probe advantage is not attributable to sampling noise\. Across all 19 students, the true teacher significantly exceeds the1/K=0\.251/K=0\.25chance baseline under a one\-sided exact binomial test \(p=3\.5×10−71p=3\.5\\times 10^\{\-71\}to3\.9×10−1213\.9\\times 10^\{\-121\}\)\. Because the test evaluates a pre\-specified teacher rather than selecting the best ofKKcandidates, no multiple\-comparison correction is required\. As an exact test, it also avoids distributional assumptions and does not require normality checks or Wilcoxon\-style alternatives\.

We additionally ablate how many teacher\-generated examples are needed for the reference\-based signal to emerge\. Starting from the same controlled setup above, we vary the number of training examples used to distill the student while keeping the probing and candidate teacher pool fixed\. Checkpointing a Qwen\-2\.5\-1\.5B student every 16 training examples and re\-running reference\-based MIA at each stage—using the base Qwen\-2\.5\-1\.5B as the reference model—we find the signal emerges almost immediately: after just 32 examples, the true teacher is correctly identified for all three teachers \(GPT\-OSS\-120B, Qwen\-3\-8B, and Nvidia\-Llama\-3\.3\-70B\-Instruct\), as shown in Figure[3](https://arxiv.org/html/2607.09692#S4.F3)\. The preceding checkpoint \(16 examples\) yields an identically zero reference\-normalized loss, since under our cosine schedule with 5% warmup the first step is taken at a near\-zero learning rate, leaving the student unchanged from the base reference\. Identification then remains stable through the end of training, indicating that only a few dozen teacher\-generated examples suffice for the true teacher to dominate\.

![Refer to caption](https://arxiv.org/html/2607.09692v1/x3.png)Figure 3:Reference\-based MIA after only 32 training examples \(checkpoint 2\)\. A Qwen\-2\.5\-1\.5B student distilled from each teacher \(GPT\-OSS\-120B, Llama\-3\.3\-70B\-Instruct, Qwen\-3\-8B\) on OMI prompts, probed with s1 and referenced against the base Qwen\-2\.5\-1\.5B\. In every panel the true teacher is the top\-ranked candidate, showing that the teacher signal is already identifiable after only a few dozen examples\.#### Effect of highly customized instructions in distillation detection

Toshniwalet al\.\[[26](https://arxiv.org/html/2607.09692#bib.bib4)\]introduce a specialized prompt format \(OMI COT\) that they show substantially alters Llama Instruct response style, making this a challenging setting where prompt\-induced stylistic artifacts may obscure the teacher signal\. FollowingToshniwalet al\.\[[26](https://arxiv.org/html/2607.09692#bib.bib4)\], we prompt Llama 3\.3 70B Instruct with the OMI COT template, distill students on this data, and test whether reference\-based MIA recovers the teacher using s1 as probing data \(since OMI was used for training\)\. Table[3](https://arxiv.org/html/2607.09692#S4.T3)shows that detection accuracy degrades substantially under this distribution shift\.

Table 3:Comparison of reference\-based teacher identification under highly customized instructions during distillation \(unknown to the detector\), with and without in\-context exemplars\.![Refer to caption](https://arxiv.org/html/2607.09692v1/x4.png)Figure 4:Results on a model distilled from Llama 3\.3 70B Instruct, using highly customized instructions \(unknown to the detector\)\. While the default method struggles to identify the true teacher \(left\), few\-shot prompting improves teacher identification \(right\)\.To address this, we introduce asimple few\-shot prompting strategy: for each candidate teacher, we prepend a small number of student\-generated \(prompt, response\) examples under the default template to better align teacher outputs with the student’s style\. As shown in the last row of Table[3](https://arxiv.org/html/2607.09692#S4.T3), this improves accuracy, recovering the true teacher for three of four student models \(the exception being Gemma\-3\-4B\-pt\)\. Figure[4](https://arxiv.org/html/2607.09692#S4.F4)illustrates the improved separation between the true teacher and alternatives, with additional CDFs shown in §[C](https://arxiv.org/html/2607.09692#A3)\.

§[B](https://arxiv.org/html/2607.09692#A2)reports additional ablations on reference\-model choice and proxy\-prompt mismatch\. Performance is generally robust to reference choice, although reliable identification benefits from a reasonably similar reference; for instruction\-tuned students, same\-family, earlier\-generation instruction\-tuned references are more effective than non\-instruction\-tuned ones\. Performance also remains stable when proxy prompts differ substantially from the original distillation inputs, including in cross\-domain settings such as math versus code\.

Table 4:Teacher identification accuracy \(%\) for real\-world distilled models \(OMI probe set\)\.Averaged over seven models: six R1\-Distill models plus s1\.1\-32B\.

![Refer to caption](https://arxiv.org/html/2607.09692v1/x5.png)Figure 5:Results fors1\.1\-32B, a real\-world model trained by\[[13](https://arxiv.org/html/2607.09692#bib.bib5)\]\. The true teacher model, R1, is ranked at the top by a large margin\.

### 4\.3Results on Real\-World Distilled Models

#### Results on DeepSeek\-R1 distill models and s1\.1\-32B

As reported in Table[4](https://arxiv.org/html/2607.09692#S4.T4), we find our method correctly identifies the true teacher among ten candidates for seven out of the seven evaluated models, both over OMI and s1 probing sets, showing that our method works when distillation data and training procedure are largely unknown\.

The same per\-probe test confirms this at the level of individual probes \(Table[5](https://arxiv.org/html/2607.09692#S4.T5)\): for all seven targets, on both prompt sets, DeepSeek\-R1’s per\-probe win rate is statistically significantly above the1/K1/Kchance baseline\. As in the controlled setting, the teacher is named in advance, so the test carries no best\-of\-KKselection penalty, certifying that this per\-probe lead is not sampling noise\.

Table 5:Per\-probe top\-1 significance on real\-world distilled models\.The true teacher is DeepSeek\-R1 for all seven targets\. For each, we count how often R1 ranks first across both prompt sets against the full candidate pool and test against chance with a one\-sided exact binomial test\. We report the range ofpp\-values across targets and prompt sets\. All seven targets are significant on both prompt sets\.The CDF graph fors1\.1\-32Bis provided in Figure[5](https://arxiv.org/html/2607.09692#S4.F5)for reference, showing that the true teacher \(R1\) is clearly separated from other candidates\.

Additional CDFs for the remaining real\-world models, including theX\-Coder\-SFT\-Qwen3\-8Bresults in Figure[17](https://arxiv.org/html/2607.09692#A4.F17), are provided in §[D](https://arxiv.org/html/2607.09692#A4)\.

#### Role of Reasoning Traces\.

One important takeaway from these experiments is that access to teacher reasoning traces is critical for reliable teacher identification\. As shown in the bottom section of Table[4](https://arxiv.org/html/2607.09692#S4.T4), when R1 reasoning traces are hidden and only final answers are available, teacher identification accuracy collapses to zero\. This suggests an important limitation for settings where full teacher reasoning traces are unavailable, such as closed reasoning models that expose only final answers or compressed summaries of their reasoning\.

## 5Experiment: Distillation Detection

This section evaluates how the margin\-threshold detector from §[3\.2](https://arxiv.org/html/2607.09692#S3.SS2)generalizes beyond the models used to calibrate it\. Selectingτ\\tauto maximize controlled accuracy yieldsτ=0\.067\\tau=0\.067, which attains85\.5%85\.5\\%\(65/76\)\(65/76\)on the controlled set but only53\.6%53\.6\\%\(15/28\)\(15/28\)when applied unchanged to the real\-world distilled cells—a single fixed threshold does not transfer cleanly between settings\. We therefore isolate two forms of generalization: transfer to an unseen*student*whose teacher was seen during calibration and transfer to an unseen*teacher*\. For each, we report the threshold’s accuracy and the significance of every detection\.

### 5\.1Experimental Setup

#### Data Construction\.

Each evaluation cell is\(S,\{T1,…,TK\},y\)\(S,\\\{T\_\{1\},\\ldots,T\_\{K\}\\\},y\), wherey=1y=1if the true teacher is present in the candidate pool andy=0y=0otherwise\. We use two sources:

- •Controlled models\.The 19 distilled students from §[4\.1](https://arxiv.org/html/2607.09692#S4.SS1), each evaluated with our Reference MIA method under OMI and s1 prompts with the true teacher included \(y=1y\{=\}1,K=4K\{=\}4\) or excluded \(y=0y\{=\}0,K=3K\{=\}3\), yielding19×2×2=7619\\times 2\\times 2=76evaluation cells\.
- •Real\-world distilled models\.Six DeepSeek\-R1 distilled models and s1\.1\-32B, each evaluated under both prompt datasets with the true teacher included \(y=1y\{=\}1,1010candidates\) or excluded \(y=0y\{=\}0,99candidates\), yielding7×2×2=287\\times 2\\times 2=28held\-out cells\.

#### Cross\-Validation\.

In every cross\-validation fold, we refit the detection thresholdτ\\tauusing only the training cells, selecting the value that maximizes F1 without access to any held\-out margins\. For the controlled models, we use two splits\. In*leave\-one\-student\-out*, we hold out one student and fitτ\\tauon the rest, leaving all teachers represented \(four folds\)\. In*leave\-one\-teacher\-out*, we hold out every distillation of one teacher, leaving that teacher absent during calibration \(three folds\)\. For the real\-world distilled models, all seven share DeepSeek\-R1 as their teacher, so we only perform leave\-one\-student\-out, fittingτ\\tauon six R1\-distilled students and evaluating the seventh\. For each held\-out cell, we report margin\-prediction accuracy \(δ≥τ\\delta\\geq\\tau\) and apply the detection test from §[3\.3](https://arxiv.org/html/2607.09692#S3.SS3)with the fold’sτ\\tauin place ofτ∗\\tau^\{\*\}\.

### 5\.2Results

#### Generalization to unseen students versus unseen teachers\.

Table[6](https://arxiv.org/html/2607.09692#S5.T6)breaks the controlled cross\-validation down by true teacher\. The two reasoning teachers \(GPT\-OSS\-120B and Qwen\-3\-8B\) are detected for every one of their students \(6/66/6each\) atp∗<10−20p^\{\*\}<10^\{\-20\}under both splits, whereas the non\-reasoning Llama\-3\.3\-70B\-Instruct is detected for only1/71/7students regardless of the split—its margin is too faint to clearτ\\tau\. The splits diverge in specificity, not detection: leaving a student out produces no false detections, whereas leaving a teacher out produces false detections on the reasoning teachers’ teacher\-removed cells \(4/64/6and1/61/6\), because the unseen teacher’s regime mis\-setsτ\\tau\. Overall, accuracy falls from82\.9%82\.9\\%\(63/76\)\(63/76\)—close to the85\.5%85\.5\\%in\-sample fit—under leave\-one\-student\-out to60\.5%60\.5\\%\(46/76\)\(46/76\)under leave\-one\-teacher\-out\.

Table 6:Controlled cross\-validation by true teacher\.*Detected*and*Detectionp∗p^\{\*\}*are the detection test on the true\-teacher\-present cells, using the leave\-one\-student\-out foldτ\\tau;*FP*is a substitute falsely called when the true teacher is removed, under the S\-outτ\\tau\(FP S\-out\) or T\-outτ\\tau\(FP T\-out\)\.*Acc\.*is the margin prediction \(δ≥τ\\delta\\geq\\tau\) per cell with the corresponding foldτ\\tau\. S\-out==leave\-one\-student\-out, T\-out==leave\-one\-teacher\-out\.
#### Generalization across real\-world students\.

The real\-world experiment reproduces the student\-side pattern \(Table[7](https://arxiv.org/html/2607.09692#S5.T7)\)\. Withτ\\taurefit on the other six R1\-distilled models in each fold, leave\-one\-student\-out attains92\.9%92\.9\\%\(26/28\)\(26/28\)accuracy\. The detection test calls six of seven held\-out students distilled from DeepSeek\-R1 \(both prompts,p∗p^\{\*\}between1\.2×10−261\.2\{\\times\}10^\{\-26\}and8\.0×10−38\.0\{\\times\}10^\{\-3\}\); when DeepSeek\-R1 is removed, no alternative ever clears the threshold \(p∗≥0\.078p^\{\*\}\\geq 0\.078\), giving zero false detections\. The single missed positive is s1\.1\-32B: DeepSeek\-R1 ranks first on both prompts, but its OMI margin \(0\.1050\.105\) falls below the fold\-fit threshold \(τ=0\.176\\tau=0\.176\), so OMI is non\-significant even though s1 is overwhelming \(p∗=8\.5×10−34p^\{\*\}=8\.5\{\\times\}10^\{\-34\}\), and the both\-prompts rule withholds the call\.

Table 7:Leave\-one\-student\-out \(real\-world\)\.Margin accuracy and significant detections \(both prompts, same teacher\); with DeepSeek\-R1 removed, no alternative is significant\.
#### Why teacher transfer is harder\.

The scale of the reference\-normalized signal depends far more on the teacher than on the student\. A threshold calibrated on the strong signals of the reasoning teachers is thus consistent across their students, however a non\-reasoning teacher may leave a margin too faint to clear it as shown with Llama 3\.3 70B Instruct in Table[7](https://arxiv.org/html/2607.09692#S5.T7), while a threshold pulled lower by that teacher admits false detections on the reasoning teachers’ teacher\-removed cells—the specificity loss seen under leave\-one\-teacher\-out\. Our results rest on only three controlled teachers, so the mechanism is suggestive rather than conclusive; nonetheless, the cross\-validation supports a clear conclusion: detection generalizes to unseen students when their teacher is represented during calibration, while threshold calibration is less reliable across unseen teachers\.

## 6OpenAI o1 Detection Methods

#### Method\.

We serialize eacho1output in two ways: Unicode, preserving raw UTF\-8 characters, and ASCII, escaping each non\-ASCII codepoint\. We then run our reference\-MIA method with all other hyperparameters fixed and defineΔASCII=ℒ¯ASCII−ℒ¯Unicode\\Delta\_\{\\text\{ASCII\}\}=\\overline\{\\mathcal\{L\}\}\_\{\\text\{ASCII\}\}\-\\overline\{\\mathcal\{L\}\}\_\{\\text\{Unicode\}\}, where positive values indicate that ASCII normalization raises reference\-normalized loss\. We evaluate fouro1\-distilled students trained on s1 using 100 held\-out OMI probes, and compare against real U\.S\.\-organization control models not expected to be distilled fromo1/o3\. To assess significance we reuse the statistical test \(§[3\.3](https://arxiv.org/html/2607.09692#S3.SS3)\) with the null placed at zero; additional details, controls, and wild\-model results are in §[E](https://arxiv.org/html/2607.09692#A5)\.

![Refer to caption](https://arxiv.org/html/2607.09692v1/x6.png)Figure 6:Per\-probe CDFs\.CDFs for a within\-family control and ano1\-distilled Gemma student\. The non\-distilledgemma\-3\-27b\-ptcontrol has nearly overlapping ASCII and Unicode curves, whereas theo1\-distilledgemma\-3\-4b\-ptstudent separates consistently\.TargetΔASCII\\Delta\_\{\\text\{ASCII\}\}95%95\\%CIppSig\.\\rowcolorred\!8gemma\-3\-4b\-pt\(SFT\)\+0\.0732\+0\.0732\[\+0\.064,\+0\.083\]\[\+0\.064,\+0\.083\]<10−15\{<\}10^\{\-15\}✓\\rowcolorred\!8Qwen2\.5\-1\.5B\(SFT\)\+0\.0594\+0\.0594\[\+0\.050,\+0\.069\]\[\+0\.050,\+0\.069\]<10−15\{<\}10^\{\-15\}✓\\rowcolorred\!8Qwen2\.5\-3B\(SFT\)\+0\.0573\+0\.0573\[\+0\.048,\+0\.067\]\[\+0\.048,\+0\.067\]<10−15\{<\}10^\{\-15\}✓\\rowcolorred\!8Llama\-3\.2\-3B\-Instruct\(SFT\)\+0\.0647\+0\.0647\[\+0\.055,\+0\.075\]\[\+0\.055,\+0\.075\]<10−15\{<\}10^\{\-15\}✓\\rowcolorblue\!8gemma\-3\-12b\-it\+0\.0373\+0\.0373\[\+0\.028,\+0\.047\]\[\+0\.028,\+0\.047\]5\.0×10−115\.0\{\\times\}10^\{\-11\}✓\\rowcolorblue\!8gemma\-3\-27b\-it\+0\.0072\+0\.0072\[−0\.001,\+0\.015\]\[\-0\.001,\+0\.015\]2\.7×10−22\.7\{\\times\}10^\{\-2\}✓\\rowcolorblue\!8gemma\-3\-27b\-pt\+0\.0070\+0\.0070\[\+0\.003,\+0\.011\]\[\+0\.003,\+0\.011\]5\.1×10−45\.1\{\\times\}10^\{\-4\}✓\\rowcolorblue\!8gemma\-2\-9b\+0\.0016\+0\.0016\[−0\.005,\+0\.008\]\[\-0\.005,\+0\.008\]0\.280\.28✗\\rowcolorblue\!8Llama\-3\.3\-70B\+0\.0007\+0\.0007\[−0\.005,\+0\.006\]\[\-0\.005,\+0\.006\]0\.660\.66✗\\rowcolorblue\!8Llama\-3\.1\-8B−0\.0024\-0\.0024\[−0\.007,\+0\.002\]\[\-0\.007,\+0\.002\]0\.630\.63✗\\rowcolorblue\!8Llama\-3\.1\-70B−0\.0050\-0\.0050\[−0\.008,−0\.002\]\[\-0\.008,\-0\.002\]1\.001\.00✗\\rowcolorblue\!8Llama\-3\.1\-8B\-Inst\.−0\.0078\-0\.0078\[−0\.017,\+0\.001\]\[\-0\.017,\+0\.001\]0\.930\.93✗\\rowcolorblue\!8gemma\-2\-9b\-it−0\.0733\-0\.0733\[−0\.105,−0\.043\]\[\-0\.105,\-0\.043\]1\.001\.00✗Table 8:ASCII vs\. Unicode serialization with significance\.Targets ranked byΔASCII\\Delta\_\{\\text\{ASCII\}\}; positive values indicate higher loss under ASCII normalization\. We reportΔASCII\\Delta\_\{\\text\{ASCII\}\}with95%95\\%bootstrap CIs; ✓ denotes statistical significance\. All fouro1\-distilled students are significant, as are three Gemma\-3 controls—but with much smaller gaps\. Red:o1\-distilled positives; blue: controls\.
#### Results\.

Figure[6](https://arxiv.org/html/2607.09692#S6.F6)and Table[8](https://arxiv.org/html/2607.09692#S6.T8)show thato1\-distilled students have consistently positive ASCII gaps, while controls mostly remain near zero\. The CDFs show the same distributional effect: the non\-distilledgemma\-3\-27b\-ptcontrol has nearly overlapping ASCII and Unicode curves, whereas for theo1\-distilledgemma\-3\-4b\-ptstudent the Unicode curve lies consistently above the ASCII curve\. This ordering—Unicode assigned higher membership likelihood than ASCII for the distilled model but not the control—may itself serve as an additional visual indicator of distillation, mirroring the positiveΔASCII\\Delta\_\{\\text\{ASCII\}\}gap\. To confirm these gaps are not sampling noise, we test each per\-probe difference against zero\.666Because Unicode is fixed as the reference serialization in advance, this is a single pre\-specified directional comparison \(K=2K=2\), so we test against zero rather than the calibrated marginτ⋆\\tau^\{\\star\}and need no candidate\-selection \(Bonferroni\) correction\.All fouro1\-distilled students have significantly positive gaps \(ΔASCII∈\[0\.0573,0\.0732\]\\Delta\_\{\\text\{ASCII\}\}\\in\[0\.0573,0\.0732\];p<10−15p<10^\{\-15\}\)\. Only the three Gemma\-3 controls also reach significance, but with much smallerΔASCII\\Delta\_\{\\text\{ASCII\}\}gaps \(gemma\-3\-12b\-it\+0\.0373\+0\.0373;gemma\-3\-27b\-it\+0\.0072\+0\.0072;gemma\-3\-27b\-pt\+0\.0070\+0\.0070\); the two Gemma\-2 controls do not\. With only100100probes, app\-value can resolve even a tiny positiveΔASCII\\Delta\_\{\\text\{ASCII\}\}from zero, so significance alone may be a weak criterion here; what matters is the size ofΔASCII\\Delta\_\{\\text\{ASCII\}\}itself—the ASCII\-minus\-Unicode difference in reference\-normalized loss\. By that measure the distilled and control gaps are clearly separated: the smallest distilled gap \(\+0\.0573\+0\.0573\) exceeds the largest control gap \(\+0\.0373\+0\.0373\) by over50%50\\%and the27bgaps by roughly8×8\\times, and all four distilled models’ confidence intervals lie entirely above every control model’s interval\. Thus, Unicode features inherited fromo1can serve as a complementary membership signal under controlled distillation\. However, because this diagnostic uses only100100OMI probes from a single prompt set, we view it as a cue rather than standalone evidence\.

## 7Open Questions

We explore the applicability of reference\-based MIA to open\-ended settings where it is unknown whether a model was distilled from a third\-party model\. Specifically, we use reference\-based MIA to probe three open\-ended attribution questions:

1. 1\.IsQwQ\-32Bpossibly distilled from Llama, o1/o3, Claude, or DeepSeek\-R1?
2. 2\.IsGPT\-OSSpossibly distilled from Llama, QwQ, o1/o3, Claude, or DeepSeek\-R1?
3. 3\.IsDeepSeek\-R1possibly distilled from Llama, QwQ, o1/o3, or Claude?

#### Disclaimer\.

Our validation relies on several simplifying assumptions—single\-teacher, SFT\-style final\-stage distillation—and limited experimental evaluations\. Real\-world models may instead be built on multiple teachers across multiple training stages\. Moreover, the candidate teacher models we consider are themselves likely deeply connected, often building on one another through complex dependency chains\[[2](https://arxiv.org/html/2607.09692#bib.bib34)\]\. Therefore, the results in this section should be interpreted as a preliminary exploration rather than evidence sufficient to support definitive conclusions\.

#### Candidate pools and references\.

For each target, we use the same candidate\-teacher pool as in our models\-in\-the\-wild experiments \(§[4\.3](https://arxiv.org/html/2607.09692#S4.SS3)\), excluding near\-variants of the target itself to avoid trivial same\-family matches\. Concretely, we excludeGPT\-OSS\-120Bwhen evaluatingGPT\-OSS\-20B, and excludeQwQ\-32B\-Previewwhen evaluatingQwQ\-32B\. We use s1 and OpenMathInstruct \(OMI\) as probing datasets throughout\. For each experiment, we include one representative figure; additional plots and detailed statistical analyses are provided in §[F](https://arxiv.org/html/2607.09692#A6)\.

![Refer to caption](https://arxiv.org/html/2607.09692v1/x7.png)Figure 7:Our reference\-based method applied toQwQ\-32B, usingQwQ\-32B\-Previewas a reference\. Results may suggestQwQ\-32Bwas distilled from R1 after the release ofQwQ\-32B\-Preview\.
#### QwQ\-32B\.

We evaluateQwQ\-32BusingQwQ\-32B\-Previewas the reference model\. As shown in Figure[7](https://arxiv.org/html/2607.09692#S7.F7),DeepSeek\-R1ranks highest across both probing sets, with a pronounced gap from the next candidate\. Combined with the release timeline—QwQ\-32B\-Previewin November 2024,DeepSeek\-R1in January 2025, andQwQ\-32Bin March 2025—this raises the possibility thatDeepSeek\-R1outputs were used to trainQwQ\-32Bfollowing the release ofQwQ\-32B\-Preview\.

![Refer to caption](https://arxiv.org/html/2607.09692v1/x8.png)Figure 8:Our reference\-based method and ASCII\-Unicode diagnostic applied toDeepSeek\-R1usingDeepSeek\-MoE\-16Bas a reference\.
#### DeepSeek R1\.

We evaluateDeepSeek\-R1, usingDeepSeek\-MoE\-16Bas the reference model\. Results are reported in Figure[8](https://arxiv.org/html/2607.09692#S7.F8)\. Our reference\-based method rankso1andQwQ\-32B\-Previewnear the top, raising the possibility thatDeepSeek\-R1was influenced by or distilled fromo1orQwQ\-32B\-Preview\.

In addition, theo1diagnostic in Figure[8](https://arxiv.org/html/2607.09692#S7.F8)\(right\) shows a clear Unicode–ASCII gap\. We test this gap one\-sided against zero \(95%95\\%bootstrap CI over100100probes; Wilcoxon, as the differences are non\-normal\): it isδUni−ASCII=\+0\.937\\delta\_\{\\text\{Uni\}\-\\text\{ASCII\}\}=\+0\.937\(\[\+0\.746,\+1\.141\]\[\+0\.746,\+1\.141\],p=8\.5×10−16p=8\.5\\times 10^\{\-16\}; Table[16](https://arxiv.org/html/2607.09692#A6.T16)\), confirming the separation is resolvable rather than a sampling artifact\. This suggests possibleo1\-style signal inDeepSeek\-R1\.

![Refer to caption](https://arxiv.org/html/2607.09692v1/x9.png)Figure 9:Our reference\-based method and ASCII\-Unicode diagonistic applied toGPT\-OSS\-120BusingGPT\-2\-XLas the reference\.
#### GPT\-OSS\-20B & GPT\-OSS\-120B\.

We evaluateGPT\-OSS\-20B & GPT\-OSS\-120B\. UnlikeQwQandDeepSeek\-R1,GPT\-OSSdoes not have a suitable same\-family checkpoint that can serve as a natural reference model\. We therefore useGPT\-2 XL, an older open\-source LM released by OpenAI, while noting that it is unlikely to be a reliable reference given the substantial temporal and architectural gap\. We include this evaluation primarily to illustrate how our method behaves in the absence of an appropriate reference model\. Results with alternative reference models are reported in §[F](https://arxiv.org/html/2607.09692#A6)\.

As shown in Figure[9](https://arxiv.org/html/2607.09692#S7.F9),DeepSeek\-R1andQwQ\-32B\-Previeware ranked highest\. However, these results are difficult to interpret, as our earlier analyses suggest thatDeepSeek\-R1andQwQ\-32B\-Previewmay themselves be influenced byo1, which is also a plausible teacher forGPT\-OSS\.777Note thato1is included in our candidate teacher models but ranks low\. However, this does not rule out the possibility thatGPT\-OSSwas influenced byo1, becauseo1does not expose its original thinking traces, and this systematically underestimates the likelihood assigned too1\.An additionalo1diagnostic \(Figure[9](https://arxiv.org/html/2607.09692#S7.F9), right\) shows that the Unicode CDF lies consistently above the ASCII CDF, and this gap is statistically resolvable across all reference models forGPT\-OSS\(Table[16](https://arxiv.org/html/2607.09692#A6.T16)\)\. Together, these observations raise the possibility thatGPT\-OSSis influenced by the sameo1/DeepSeek\-R1/QwQlineage\. However, becauseGPT\-OSSlacks a suitable reference model, this conclusion remains highly uncertain\. We leave a more systematic analysis to future work that relaxes the assumption that an appropriate reference model is available\.

## 8Conclusion

We introduce reference based distillation detection: identifying the teacher used to train a model given an earlier\-generation checkpoint from the same lineage\. By comparing normalized likelihood differences between a student and reference model, our approach identifies candidate teachers whose outputs are preferentially aligned with the student\. Across controlled and real\-world settings—including DeepSeek\-R1 distills and s1\.1—our method consistently identifies the true teacher and outperforms similarity\-based baselines, while remaining effective under cross\-domain evaluation\. Overall, our work provides an initial step toward practical auditing of distillation in modern LLMs\.

#### Limitations\.

Our evaluation targets a canonical setting—single\-teacher, SFT\-style, final\-stage distillation—and does not address multi\-teacher or multi\-stage pipelines\. Identification quality depends on having a suitable reference checkpoint, which is not always available, as illustrated byGPT\-OSS\. Our signal also performs well with reasoning traces of candidate teachers, so the method is most reliable for teachers that expose full traces\. A further question we do not resolve concerns*rewriting*pipelines: if data is generated by teacherAAand then paraphrased or rewritten by teacherBBbefore training, it is unclear which model our method would attribute the student to—AA,BB, some mixture, or neither\. The detected signal could plausibly track either the originating model or the rewriter depending on how much stylistic and distributional structure survives the rewrite, and we leave a systematic study of such to future work\.

## Acknowledgement

We thank David Wagner and the SM group members for valuable discussion and feedback\. This work was supported in part by gifts from OpenAI, Ai2, and Apple\.

## References

- \[1\]M\. Abdin, J\. Aneja, H\. Behl, S\. Bubeck, R\. Eldan, S\. Gunasekar, M\. Harrison, R\. J\. Hewett, M\. Javaheripi, P\. Kauffmann,et al\.\(2024\)Phi\-4 technical report\.arXiv preprint arXiv:2412\.08905\.Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px1.p1.1)\.
- \[2\]\(2026\)Which models are our models built on? auditing invisible dependencies in modern llms\.External Links:2606\.12385,[Link](https://arxiv.org/abs/2606.12385)Cited by:[§7](https://arxiv.org/html/2607.09692#S7.SS0.SSS0.Px1.p1.1)\.
- \[3\]Anthropic\(2026\)Detecting and preventing distillation attacks\.External Links:[Link](https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks)Cited by:[§1](https://arxiv.org/html/2607.09692#S1.p1.1),[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px2.p1.1)\.
- \[4\]J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton\(2021\)Program synthesis with large language models\.External Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732)Cited by:[footnote 10](https://arxiv.org/html/2607.09692#footnote10)\.
- \[5\]N\. Carlini, S\. Chien, M\. Nasr, S\. Song, A\. Terzis, and F\. Tramer\(2022\)Membership inference attacks from first principles\.InIEEE Symposium on Security and Privacy,Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px4.p1.1)\.
- \[6\]Gemma Team\(2025\)Gemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.External Links:[Link](https://arxiv.org/abs/2503.19786)Cited by:[Appendix A](https://arxiv.org/html/2607.09692#A1.SS0.SSS0.Px1.p1.1),[Appendix A](https://arxiv.org/html/2607.09692#A1.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.09692#S4.SS1.SSS0.Px1.p2.2)\.
- \[7\]G\. Hinton, O\. Vinyals, and J\. Dean\(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§1](https://arxiv.org/html/2607.09692#S1.p1.1),[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px1.p1.1)\.
- \[8\]X\. Jiao, Y\. Yin, S\. Lifeng, X\. Jiang, X\. Chen, L\. Li, F\. Wang, and Q\. Liu\(2020\)TinyBERT: distilling bert for natural language understanding\.InFindings of the Association for Computational Linguistics: EMNLP 2020,Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px1.p1.1)\.
- \[9\]Y\. Kim and A\. M\. Rush\(2016\)Sequence\-level knowledge distillation\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]P\. Maini, H\. Jia, N\. Papernot, and A\. Dziedzic\(2024\)LLM dataset inference: did you train on my dataset?\.External Links:2406\.06443,[Link](https://arxiv.org/abs/2406.06443)Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px4.p1.1)\.
- \[11\]MetaLlama 3\.3 \| model cards and prompt formats\.Note:[https://www\.llama\.com/docs/model\-cards\-and\-prompt\-formats/llama3\_3/](https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_3/)Model card for Llama 3\.3 70B Instruct, accessed 2026\-04\-05Cited by:[Appendix A](https://arxiv.org/html/2607.09692#A1.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.09692#S4.SS1.SSS0.Px1.p2.2)\.
- \[12\]MetaMeta\-llama/llama\-3\.2\-3b\-instruct\.Note:[https://huggingface\.co/meta\-llama/Llama\-3\.2\-3B\-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)Hugging Face model card, accessed 2026\-04\-05Cited by:[Appendix A](https://arxiv.org/html/2607.09692#A1.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.09692#S4.SS1.SSS0.Px1.p2.2)\.
- \[13\]N\. Muennighoff, Z\. Yang, W\. Shi, X\. L\. Li, L\. Fei\-Fei, H\. Hajishirzi, L\. Zettlemoyer, P\. Liang, E\. Candès, and T\. Hashimoto\(2025\)S1: simple test\-time scaling\.arXiv preprint arXiv:2501\.19393\.Cited by:[Appendix A](https://arxiv.org/html/2607.09692#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2607.09692#S1.p1.1),[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px1.p1.1),[Figure 5](https://arxiv.org/html/2607.09692#S4.F5.3),[Figure 5](https://arxiv.org/html/2607.09692#S4.F5.3.3.2),[§4\.1](https://arxiv.org/html/2607.09692#S4.SS1.SSS0.Px1.p2.2)\.
- \[14\]N\. News\(2026\)Google says attackers used 100,000\+ prompts to try to clone ai chatbot gemini\.External Links:[Link](https://www.nbcnews.com/tech/security/google-gemini-hit-100000-prompts-cloning-attempt-rcna258657)Cited by:[§1](https://arxiv.org/html/2607.09692#S1.p1.1),[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px1.p2.1)\.
- \[15\]OpenAI\(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.External Links:[Link](https://arxiv.org/abs/2508.10925)Cited by:[Appendix A](https://arxiv.org/html/2607.09692#A1.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.09692#S4.SS1.SSS0.Px1.p2.2)\.
- \[16\]L\. Pan, A\. Liu, S\. Huang, Y\. Lu, X\. Hu, L\. Wen, I\. King, and P\. S\. Yu\(2025\)Can llm watermarks robustly prevent unauthorized knowledge distillation?\.External Links:2502\.11598,[Link](https://arxiv.org/abs/2502.11598)Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px2.p2.1)\.
- \[17\]C\. Philipp\(2026\)From prompt to clone: copyright challenges in ai model distillation\.UC Law Science and Technology Journal17\(1\),pp\. 49\.Cited by:[§4\.1](https://arxiv.org/html/2607.09692#S4.SS1.SSS0.Px1.p1.1)\.
- \[18\]Qwen Team, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu\(2025\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.External Links:[Link](https://arxiv.org/abs/2412.15115)Cited by:[Appendix A](https://arxiv.org/html/2607.09692#A1.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2607.09692#S4.SS1.SSS0.Px1.p2.2)\.
- \[19\]Qwen Team\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[Appendix A](https://arxiv.org/html/2607.09692#A1.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2607.09692#S4.SS1.SSS0.Px1.p2.2)\.
- \[20\]T\. Sander, P\. Fernandez, A\. Durmus, M\. Douze, and T\. Furon\(2024\)Watermarking makes language models radioactive\.External Links:2402\.14904,[Link](https://arxiv.org/abs/2402.14904)Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px2.p2.1)\.
- \[21\]V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf\(2019\)DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.arXiv preprint arXiv:1910\.01108\.Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px1.p1.1)\.
- \[22\]D\. Seetharaman and F\. Arámburo\(2026\)OpenAI says china’s deepseek trained its ai by distilling us models, memo shows\.External Links:[Link](https://www.reuters.com/world/china/openai-accuses-deepseek-distilling-us-models-gain-advantage-bloomberg-news-2026-02-12/)Cited by:[§1](https://arxiv.org/html/2607.09692#S1.p1.1),[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px1.p2.1)\.
- \[23\]Q\. Shi, A\. Y\. Zheng, Q\. Song, and R\. A\. Yeh\(2025\)Knowledge distillation detection for open\-weights models\.arXiv preprint arXiv:2510\.02302\.Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2607.09692#S3.SS1.p1.4)\.
- \[24\]R\. Shokri, M\. Stronati, C\. Song, and V\. Shmatikov\(2017\)Membership inference attacks against machine learning models\.InIEEE Symposium on Security and Privacy,Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px4.p1.1)\.
- \[25\]R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto\(2023\)Alpaca: a strong, replicable instruction\-following model\.Stanford Center for Research on Foundation Models\. https://crfm\. stanford\. edu/2023/03/13/alpaca\. html3\(6\),pp\. 7\.Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px1.p1.1)\.
- \[26\]S\. Toshniwal, W\. Du, I\. Moshkov, B\. Kisacanin, A\. Ayrapetyan, and I\. Gitman\(2024\)OpenMathInstruct\-2: accelerating ai for math with massive open\-source instruction data\.arXiv preprint arXiv:2410\.01560\.Cited by:[Appendix A](https://arxiv.org/html/2607.09692#A1.SS0.SSS0.Px3.p1.1),[Appendix E](https://arxiv.org/html/2607.09692#A5.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2607.09692#S1.p1.1),[§4\.1](https://arxiv.org/html/2607.09692#S4.SS1.SSS0.Px1.p2.2),[§4\.2](https://arxiv.org/html/2607.09692#S4.SS2.SSS0.Px1.p1.1)\.
- \[27\]Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi\(2022\)Self\-instruct: aligning language models with self\-generated instructions\.arXiv preprint arXiv:2212\.10560\.Cited by:[§1](https://arxiv.org/html/2607.09692#S1.p1.1)\.
- \[28\]L\. Watson, C\. Guo, G\. Cormode, and A\. Sablayrolles\(2022\)On the importance of difficulty calibration in membership inference attacks\.External Links:2111\.08440,[Link](https://arxiv.org/abs/2111.08440)Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px4.p1.1),[§3\.1](https://arxiv.org/html/2607.09692#S3.SS1.SSS0.Px1.p2.7)\.
- \[29\]Y\. Xie, J\. Song, H\. Wang, and M\. Song\(2025\)Training data provenance verification: did your model use synthetic data from my generative model for training?\.External Links:2503\.09122,[Link](https://arxiv.org/abs/2503.09122)Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px3.p1.1)\.
- \[30\]H\. Zhang, H\. K\. Choi, S\. Li, and H\. Wei\(2025\)Detecting distillation data from reasoning models\.External Links:2510\.04850,[Link](https://arxiv.org/abs/2510.04850)Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px2.p3.1)\.
- \[31\]L\. Zhang, H\. Wu, L\. Zhang, F\. Xu, J\. Cao, F\. Li, and B\. Niu\(2024\)Training data attribution: was your model secretly trained on data created by mine?\.External Links:2409\.15781,[Link](https://arxiv.org/abs/2409.15781)Cited by:[§2](https://arxiv.org/html/2607.09692#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AExperimental Details

#### Base student models \(S0S\_\{0\}\)

We consider Llama\-3\.2\-3B\-Instruct\[[12](https://arxiv.org/html/2607.09692#bib.bib21)\], Gemma\-3\-4B\-PT\[[6](https://arxiv.org/html/2607.09692#bib.bib22)\], and Qwen2\.5\-1\.5B and Qwen2\.5\-3B\[[18](https://arxiv.org/html/2607.09692#bib.bib23)\]as base student models\.

#### Teacher models \(𝒯\\mathcal\{T\}\)

Our teacher candidate set includes Llama\-3\.3\-70B\-Instruct\[[11](https://arxiv.org/html/2607.09692#bib.bib24)\],888We use the quantized version fromnvidia/Llama\-3\.3\-70B\-Instruct\-NVFP8\.Qwen3\-8B\[[19](https://arxiv.org/html/2607.09692#bib.bib25)\], Gemma\-3\-27B\[[6](https://arxiv.org/html/2607.09692#bib.bib22)\], and GPT\-OSS\-120B\[[15](https://arxiv.org/html/2607.09692#bib.bib26)\]\.

These four models together form the pool of candidate teachers for the teacher identification task, allowing each to act as both a positive and a negative candidate\. Consequently, the task is formulated as a four\-way classification task\. To prevent our task from becoming trivial due to substantial variation in teacher output lengths, we standardize teacher outputs prior to distillation by truncating each output to at most 2,048 tokens\.

#### Inputs \(𝒳\\mathcal\{X\}\)

We use two open\-source prompt datasets: OpenMathInstruct\-2 \(OMI\-2\), a dataset of math problems and reasoning questions, and s1, another set of reasoning\-heavy prompts used for test\-time scaling\[[26](https://arxiv.org/html/2607.09692#bib.bib4),[13](https://arxiv.org/html/2607.09692#bib.bib5)\]\.

We create distilled student models by querying each teacher with input prompts, and fine\-tuning each student on the resulting teacher\-generated data\. In total, this setup produces 4 base student models×\\times3 teacher models×\\times2 prompt datasets, yielding 24 distilled models\. We then retain models only when distillation leads to performance improvements, outperforming their corresponding base models,999We use GSM8K and MATH 500 to evaluate whether distillation improves performance\. Detailed results are reported in Table[9](https://arxiv.org/html/2607.09692#A1.T9)\.\.resulting in 19 models\.

#### Baseline detection methods

We consider the following baselines\. \(1\)Raw likelihood, which averages the token\-level log\-likelihood ofTk​\(xi\)T\_\{k\}\(x\_\{i\}\)underSS\. \(2\)ZLIB\-normalized likelihood, which normalizes likelihood using compression\-based length estimates\. \(3\)Min\-kk% probability, which focuses on the lowest\-probability tokens in the sequence\. \(4\)Similarity\-based scoring, which compares teacher and student outputs using lexical overlap \(e\.g\.,nn\-gram Jaccard\) and embedding\-based similarity\. Finally, we compare to our primary method,reference\-based MIA, which normalizes likelihood using a reference model\. We also evaluate areverse MIAvariant, where we instead generate outputs from the studentS​\(xi\)S\(x\_\{i\}\)and score them under each candidate teacher modelTkT\_\{k\}\. Results are shown in Table[13](https://arxiv.org/html/2607.09692#A2.T13)\.

#### Evaluation Metrics

We evaluate teacher identification using two complementary decision rules\. First, we report a sorted per\-sample score: for each candidate teacher, we sort its alignment scores over the probe set and compare candidates at matched ranks, measuring how often the true teacher obtains the highest alignment score \(Per\-sample\)\. Formally, letting𝐟Tj,r↓\\mathbf\{f\}^\{\\downarrow\}\_\{T\_\{j\},r\}denote therr\-th largest alignment score for teacherTjT\_\{j\}over𝒳^\\hat\{\\mathcal\{X\}\}:

T^sorted=arg⁡maxTk∈𝒯⁡1N​∑r=1N𝟏​\[Tk=arg⁡maxTj∈𝒯⁡𝐟Tj,r↓\]\.\\hat\{T\}\_\{\\mathrm\{sorted\}\}=\\arg\\max\_\{T\_\{k\}\\in\\mathcal\{T\}\}\\frac\{1\}\{N\}\\sum\_\{r=1\}^\{N\}\\mathbf\{1\}\\\!\\left\[T\_\{k\}=\\arg\\max\_\{T\_\{j\}\\in\\mathcal\{T\}\}\\mathbf\{f\}^\{\\downarrow\}\_\{T\_\{j\},r\}\\right\]\.This captures whether the true teacher is favored distributionally, rather than requiring it to win on the same prompt\. Second, we report aggregate identification accuracy by applying both decision rules at the model level: selecting the teacher with the highest mean alignment score,

T^mean=arg⁡maxTk∈𝒯⁡1N​∑i=1N𝐟​\(Tk​\(xi\),S\),\\hat\{T\}\_\{\\mathrm\{mean\}\}=\\arg\\max\_\{T\_\{k\}\\in\\mathcal\{T\}\}\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{f\}\(T\_\{k\}\(x\_\{i\}\),S\),and selecting the teacher that wins most often under the sorted per\-sample comparison\. We then report the higher accuracy between these two aggregate decision rules \(Agg\.\)\.

We report the models used in the main paper for our controlled experiments \(Table[9](https://arxiv.org/html/2607.09692#A1.T9)\), including only settings where distillation improves utility over the base model on both GSM8K and MATH500\. When using our highly customized prompt template \(OMI COT\), we include all distilled models, observing gains in at least one of GSM8K or MATH500\.

#### Distillation hyperparameters

For each \(student, teacher, prompt dataset\) combination, we fine\-tune the base studentS0S\_\{0\}on the teacher\-generated dataset𝒟T\\mathcal\{D\}\_\{T\}using supervised fine\-tuning\. We use identical hyperparameters across all controlled distillation runs: 3 epochs, learning rate1​e−51\\mathrm\{e\}\{\-5\}with cosine schedule and 5% warmup, per\-device batch size 4 with gradient accumulation 4 \(effective batch size 16\), block size 4,096, bf16 precision, and gradient checkpointing\. Loss is computed on response tokens only; prompt tokens are masked with−100\-100\.

For Llama\-3\.2\-3B\-Instruct, we apply the model’s native chat template \(apply\_chat\_template\) to format each \(question, response\) pair, and identify the prompt span via the prompt\-only tokenization withadd\_generation\_prompt=Trueso that supervision is restricted to the assistant turn\. For the non\-instruct base students \(Gemma\-3\-4B\-PT, Qwen\-2\.5\-1\.5B, Qwen\-2\.5\-3B\), we use a simpleProblem:\\n\{question\}\\n\\nSolution:\\ntemplate\. Teacher responses are truncated to 2,048 tokens prior to training to prevent variation in teacher output length making our detection task easier, and the same truncation is applied when computing reference\-based MIA scores at evaluation time\.

#### Compute resources

We ran all training and evaluation on H200 GPUs\. Each controlled distillation run uses 2×\\timesH200, as do all reference\-based MIA evaluations and baselines \(similarity\-based, logit\-based, reverse MIA, ablations, and the o1 ASCII\-vs\-Unicode diagnostic\)\. Larger candidate teachers were run on the same 2×\\timesH200 setup, except for DeepSeek\-R1, which we ran locally on 8×\\timesH200 due to its size\. Outputs from API\-only models were generated through their respective APIs and did not require local GPUs\.

We additionally report descriptive statistics for teacher output lengths in Table[10](https://arxiv.org/html/2607.09692#A1.T10)\. These statistics provide context for our controlled distillation setup, since teacher models differ substantially in output verbosity, especially when full reasoning traces are included\. For models with explicit reasoning traces, we also report final\-output\-only lengths to separate reasoning verbosity from the final answer length\.

Table 9:Student model training results for the controlled distillation setups\.Table 10:Average output length in words across teacher models\.We report the average number of words generated by each model under s1 prompts and OMI prompts\.

## Appendix BAdditional Experimental Results

Table 11:Effect of the reference modelRRon teacher identification, when the base student model is Llama 3\.2 3B Instruct\. We note that using GPT\-OSS\-120B as a teacher did not lead to major utility, but we include it for diversity in this ablation\.Table 12:Impact of the choice of proxy input prompts \(𝒳^\\hat\{\\mathcal\{X\}\}\)\. ‘Identical’ indicates𝒳\\mathcal\{X\}and𝒳^\\hat\{\\mathcal\{X\}\}are identical; ‘Similar domain’ indicates𝒳\\mathcal\{X\}and𝒳^\\hat\{\\mathcal\{X\}\}are not identical but are from similar domains; and ‘Different domain’ indicates𝒳\\mathcal\{X\}and𝒳^\\hat\{\\mathcal\{X\}\}are from substantially different domains\. Detection accuracy is relatively stable across all settings\.### B\.1Ablations on the choice of reference model \(RR\)

We ablate the choice of reference model R to assess its impact on teacher identification \(Table[11](https://arxiv.org/html/2607.09692#A2.T11)\)\. Using Llama 3\.2\-3B\-Instruct students, we varyRRacross related base and instruction\-tuned models\. Overall, performance is reasonably robust to the choice ofRR; however, using base \(non\-instruction\-tuned\) models degrades accuracy\. This suggests that although the method is reasonably stable, matching the instruction\-tuning of the student is important for reliable identification\.

### B\.2Ablations on proximity in input prompts \(𝒳^\\hat\{\\mathcal\{X\}\}\)

To quantify the effect of input prompt approximation—i\.e\., the mismatch between the true distillation prompts𝒳\\mathcal\{X\}and the prompts that a detector has access to𝒳^\\hat\{\\mathcal\{X\}\}—we consider three settings:

1. 1\.Identical:𝒳\\mathcal\{X\}and𝒳^\\hat\{\\mathcal\{X\}\}are identical, e\.g\., both are s1 prompts, or both are OMI prompts\.
2. 2\.Similar\-domain:𝒳\\mathcal\{X\}and𝒳^\\hat\{\\mathcal\{X\}\}are not identical but are from similar domains, e\.g\., \(𝒳=\\mathcal\{X\}=s1,𝒳^=\\hat\{\\mathcal\{X\}\}=OMI\), or \(𝒳=\\mathcal\{X\}=OMI,𝒳^=\\hat\{\\mathcal\{X\}\}=s1\)\. This was the default setting in our main experiments\.
3. 3\.Different\-domain:𝒳\\mathcal\{X\}and𝒳^\\hat\{\\mathcal\{X\}\}are from substantially different domains, e\.g\.,𝒳\\mathcal\{X\}is s1 or OMI \(math\), while𝒳^\\hat\{\\mathcal\{X\}\}consists of prompts for code generation\.101010We took input prompts from MBPP\[[4](https://arxiv.org/html/2607.09692#bib.bib27)\]

We follow the same setup as in the main experiments: models are distilled using four base students, four candidate teachers, and two prompt sets \(s1 or OMI\), and the detection task is to identify the true teacher among four candidates\. Table[12](https://arxiv.org/html/2607.09692#A2.T12)reports the results across these three proxy\-input settings\. Our method remains accurate even when the detector uses prompts from a different domain \(code probing data\), suggesting that the teacher\-identification signal is not limited to the exact prompt distribution used during distillation\.

### B\.3Ablations for Reverse MIA methods

Table 13:Teacher identification accuracy \(%\) for reverse logit\-based baselines\. Accuracy is broken down by true teacher model, including both aggregated and per\-sample metrics\. Reverse likelihood variants perform well for Llama 3\.3 70B Instruct but fail to reliably identify Qwen\-3\-8B and GPT\-OSS\-120B, while our reference\-based method remains accurate across teachers\.

## Appendix CDetecting Distillation with a Highly Customized Prompt template

We provide additional visualization of the few\-shot prompting strategy described in the main paper \(Table[3](https://arxiv.org/html/2607.09692#S4.T3)\)\. While the table reports aggregate detection accuracy, the plots below illustrate how few\-shot prompting affects the distribution of alignment scores across candidate teachers\.

Each subplot corresponds to a different distilled student model, and shows how candidate teachers behave when prompted with a small number of student\-generated examples\. We observe that few\-shot prompting generally sharpens the separation between the true teacher and alternative candidates, improving identification performance\. However, for the Gemma\-3\-4B\-pt student, the separation remains weaker, consistent with the failure case observed in Table[3](https://arxiv.org/html/2607.09692#S4.T3)\.

![[Uncaptioned image]](https://arxiv.org/html/2607.09692v1/x10.png)

\(a\) Qwen\-2\.5\-1\.5B

![[Uncaptioned image]](https://arxiv.org/html/2607.09692v1/x11.png)

\(b\) Qwen\-2\.5\-3B

![[Uncaptioned image]](https://arxiv.org/html/2607.09692v1/x12.png)

\(c\) Llama\-3\.2\-3B\-Instruct

![[Uncaptioned image]](https://arxiv.org/html/2607.09692v1/x13.png)

\(d\) Gemma\-3\-4B\-pt

Figure 10:Results of prompting each candidate teacher model with few\-shot examples drawn from the outputs of a distilled student model\. Each subplot corresponds to a different student and evaluates how well candidate teachers can match the student’s behavior under in\-context learning\.

## Appendix DModel in the Wild Plots

We evaluate the DeepSeek\-R1 distill models,s1\.1\-32B, and XCoder using reference\-based membership inference \(reference\-based MIA\)\. For each distilled model, we use as the reference model the corresponding base model from which the distill model was derived prior to distillation\. We run the method on both OMI and s1 probe data, allowing us to test whether teacher\-identification signals persist across related math\-domain prompt distributions\.

Overall, our method successfully identifies the true teacher across the evaluated candidates, showing that the reference\-normalized likelihood signal remains informative even for real\-world distilled models whose exact training data and distillation pipelines are unknown\. This is encouraging because these models are not produced under our controlled setup: the prompts, filtering procedures, decoding parameters, and post\-training details are all unobserved\. Nevertheless, the true teacher generally receives the strongest alignment signal\.

We further evaluate XCoder, a code\-specialized model trained on code data, while still probing it using math data from s1 and OMI\. The method continues to show a teacher\-specific signal in this out\-of\-domain setting, suggesting that the attribution signal is not limited to models trained directly on math data\.

![[Uncaptioned image]](https://arxiv.org/html/2607.09692v1/x14.png)

Figure 11:Reference\-based MIA results forDeepSeek\-R1\-Distill\-Qwen\-1\.5B, usingQwen2\.5\-Math\-1\.5Bas the reference model\.
![[Uncaptioned image]](https://arxiv.org/html/2607.09692v1/x15.png)

Figure 12:Reference\-based MIA results forDeepSeek\-R1\-Distill\-Qwen\-7B, usingQwen2\.5\-Math\-7Bas the reference model\.
![[Uncaptioned image]](https://arxiv.org/html/2607.09692v1/x16.png)

Figure 13:Reference\-based MIA results forDeepSeek\-R1\-Distill\-Llama\-8B, usingLlama\-3\.1\-8Bas the reference model\.
![[Uncaptioned image]](https://arxiv.org/html/2607.09692v1/x17.png)

Figure 14:Reference\-based MIA results forDeepSeek\-R1\-Distill\-Qwen\-14B, usingQwen2\.5\-14Bas the reference model\.
![[Uncaptioned image]](https://arxiv.org/html/2607.09692v1/x18.png)

Figure 15:Reference\-based MIA results forDeepSeek\-R1\-Distill\-Qwen\-32B, usingQwen2\.5\-32Bas the reference model\.
![[Uncaptioned image]](https://arxiv.org/html/2607.09692v1/x19.png)

Figure 16:Reference\-based MIA results forDeepSeek\-R1\-Distill\-Llama\-70B, usingLlama\-3\.3\-70B\-Instructas the reference model\.
![[Uncaptioned image]](https://arxiv.org/html/2607.09692v1/x20.png)

Figure 17:Reference\-based MIA results onXCoder\. Although XCoder was trained on code data, we probe it using math datasets \(s1 and OMI\) to test whether teacher\-specific likelihood signals remain detectable out of domain\.

## Appendix EAdditional o1 Detection Details

#### Background: ASCII and Unicode\.

ASCII is a 7\-bit encoding covering 128 characters; Unicode assigns codepoints to over 140,000 characters spanning nearly all writing systems and mathematical notation\. ASCII is a strict subset of Unicode, so any Unicode string can be losslessly rendered as ASCII by escaping each non\-ASCII codepoint as its literal\\uXXXXsequence\.

#### Methodology\.

We generate eacho1output once and serialize it two ways: a Unicode form \(ensure\_ascii=False\) preserving raw UTF\-8 glyphs, and an ASCII form \(ensure\_ascii=True\) in which each non\-ASCII codepoint becomes its six\-character escape \(e\.g\. “•”→\\to\\u2022\)\. For each target, we run reference\-based MIA twice with all other hyperparameters fixed and defineΔASCII=ℒ¯ASCII−ℒ¯Unicode\\Delta\_\{\\text\{ASCII\}\}=\\overline\{\\mathcal\{L\}\}\_\{\\text\{ASCII\}\}\-\\overline\{\\mathcal\{L\}\}\_\{\\text\{Unicode\}\}on the reference\-normalized loss\. Under the null that the detector is not exploiting glyph\-level features,ΔASCII≈0\\Delta\_\{\\text\{ASCII\}\}\\approx 0; a positive gap indicates that ASCII normalization*raises*the target’s loss relative to the Unicode baseline—i\.e\., that the encoding contributes to the membership signal\.

#### Controlled positives and probing data\.

To establish a ground\-truth positive condition, we supervised\-fine\-tune four students—Qwen2\.5\-1\.5B,Qwen2\.5\-3B,Llama\-3\.2\-3B\-Instruct, andgemma\-3\-4b\-pt—witho1as the teacher on 1,000 questions drawn from the s1 dataset\. Because the OpenAI API exposes only a*summarized*reasoning trace, each training target is the concatenation of the summarized trace and the final answer\. The MIA reference for each student is its own pre\-SFT base checkpoint, isolating any signal acquired during distillation\. As negative controls, we use real U\.S\.\-organization models that are not expected to be distilled fromo1/o3\. We use 100 held\-out OMI questions as probing data, providing a realistic setting where the detector does not have access to the original training inputs\.

#### Additional control and wild\-model results\.

We additionally evaluate a broader set of within\-family controls, controlledo1\-distilled positives, and wild models with large observed ASCII gaps\. Table[14](https://arxiv.org/html/2607.09692#A5.T14)reports the reference model used for each target and the resultingΔASCII\\Delta\_\{\\text\{ASCII\}\}\.

TargetReferenceΔASCII\\Delta\_\{\\text\{ASCII\}\}Within\-family controls \(not distilled fromo1/o3\)Llama\-3\.1\-8B\-InstructLlama\-3\-8B\-Instruct−0\.0078\-0\.0078Llama\-3\.3\-70B\-InstructLlama\-3\.1\-70B\-Instruct\+0\.0007\+0\.0007Llama\-3\.1\-8BLlama\-3\-8B−0\.0024\-0\.0024Llama\-3\.1\-70BLlama\-3\-70B−0\.0050\-0\.0050gemma\-2\-9b\-itgemma\-1\.1\-7b\-it−0\.0733\-0\.0733gemma\-3\-12b\-itgemma\-2\-9b\-it\+0\.0373\+0\.0373gemma\-3\-27b\-itgemma\-2\-27b\-it\+0\.0072\+0\.0072gemma\-2\-9bgemma\-7b\+0\.0016\+0\.0016gemma\-3\-27b\-ptgemma\-2\-27b\+0\.0070\+0\.0070o1\-distilled positivesgemma\-3\-4b\-pt \(SFT\)gemma\-3\-4b\-pt\+0\.0732\+0\.0732Qwen2\.5\-1\.5B \(SFT\)Qwen2\.5\-1\.5B\+0\.0594\+0\.0594Qwen2\.5\-3B \(SFT\)Qwen2\.5\-3B\+0\.0573\+0\.0573Llama\-3\.2\-3B\-Instruct \(SFT\)Llama\-3\.2\-3B\-Instruct\+0\.0647\+0\.0647Models in the wildDeepSeek\-R1DeepSeek\-MoE\-16B\-Base\+0\.937\+0\.937GPT\-OSS\-120BGPT\-OSS\-20B\+0\.3207\+0\.3207GPT\-OSS\-120BGPT\-2 XL\+0\.8310\+0\.8310GPT\-OSS\-20BGPT\-2 XL\+0\.5103\+0\.5103Table 14:ASCII gapΔASCII\\Delta\_\{\\text\{ASCII\}\}on controlled and wild models\.For each target, we report the reference model used andΔASCII=ℒ¯ASCII−ℒ¯Unicode\\Delta\_\{\\text\{ASCII\}\}=\\overline\{\\mathcal\{L\}\}\_\{\\text\{ASCII\}\}\-\\overline\{\\mathcal\{L\}\}\_\{\\text\{Unicode\}\}on the reference\-normalized loss, computed on 100 OMI probes \(see §[6](https://arxiv.org/html/2607.09692#S6)\)\. Within\-family controls cluster near zero, controlledo1\-distilled positives show consistently positive gaps, andDeepSeek\-R1and bothGPT\-OSSmodels exhibit large positive gaps\.![Refer to caption](https://arxiv.org/html/2607.09692v1/x21.png)Figure 18:ASCII vs\. Unicode CDFs for non\-o1\-distilled Llama controls\.For Llama models not expected to be distilled fromo1, the ASCII and Unicode reference\-normalized loss distributions nearly overlap, indicating little to no encoding\-specific separation\.The within\-family controls—Llama and Gemma checkpoints not expected to be distilled fromo1oro3—produce ASCII gaps concentrated near zero, with most pairs falling within±0\.01\\pm 0\.01and the largest deviation at−0\.0733\-0\.0733forgemma\-2\-9b\-it\. This is also visible in Figure[18](https://arxiv.org/html/2607.09692#A5.F18): for non\-o1\-distilled Llama controls, the ASCII and Unicode CDFs lie almost on top of each other, showing no substantial difference in reference\-normalized loss under the two serializations\. This is consistent with the hypothesis that noo1\-specific encoding signal is present in models not exposed too1during training, and matches the pattern reported in the main text\.

In contrast, the controlledo1\-distilled positives all exhibit positive gaps \(\+0\.0573\+0\.0573to\+0\.0732\+0\.0732\), while the real\-world models exhibit much larger positive gaps\.DeepSeek\-R1, withDeepSeek\-MoE\-16B\-Baseas reference, producesΔASCII=\+0\.937\\Delta\_\{\\text\{ASCII\}\}=\+0\.937\. UsingGPT\-2 XLas the reference,GPT\-OSS\-20BproducesΔASCII=\+0\.5103\\Delta\_\{\\text\{ASCII\}\}=\+0\.5103andGPT\-OSS\-120BproducesΔASCII=\+0\.8310\\Delta\_\{\\text\{ASCII\}\}=\+0\.8310; usingGPT\-OSS\-20Bas a within\-family reference forGPT\-OSS\-120Bstill yieldsΔASCII=\+0\.3207\\Delta\_\{\\text\{ASCII\}\}=\+0\.3207\. Thus, the real\-world gaps are not only positive but also substantially larger than those of our controlledo1\-distilled students, often by nearly an order of magnitude\. This mirrors the per\-probe CDF separation forGPT\-OSSandDeepSeek\-R1reported in §[7](https://arxiv.org/html/2607.09692#S7)\.

#### Surface\-Level Stylistic Markers as Complementary Signals

We additionally observe that certain teacher models exhibit highly consistent surface\-level generation habits\. Specifically, Llama\-3\.3\-70B\-Instruct responses follow a structured Markdown format using headings of the form\#\# StepNN, whichToshniwalet al\.\[[26](https://arxiv.org/html/2607.09692#bib.bib4)\]document as the characteristic “Llama CoT” format\. We similarly observe thatclaude\-3\.5\-sonnetresponses frequently contain the phrase “Let’s solve this step by step,” andQwQ\-32B\-Previewresponses frequently open with “I’ve got this problem\.” Notably,Toshniwalet al\.\[[26](https://arxiv.org/html/2607.09692#bib.bib4)\]show that a highly customized prompt template can suppress these stylistic artifacts during distillation, which may limit their utility as attribution signals\. We leave a more systematic study—including robustness to prompt variation and integration with likelihood\-based scores—to future work\.

#### Results\.

Table[15](https://arxiv.org/html/2607.09692#A5.T15)reports occurrence rates across the first 200 OMI responses per teacher\. The Llama marker is especially strong, appearing in every response with an average of 7\.26 occurrences per response, while the Claude and QwQ markers occur at nontrivial but lower rates\.

Table 15:Surface\-level stylistic marker occurrence rates across the first 200 OMI responses per teacher model\.

## Appendix FAdditional Open Question Plots

This section provides additional reference\-based MIA plots and statistical analysis for the open\-question analyses in §[7](https://arxiv.org/html/2607.09692#S7)\. These settings involve target models whose training data, prompts, instructions, filtering strategies, and distillation procedures are not publicly known, so the results should be interpreted as diagnostic rather than conclusive evidence of distillation\.

![Refer to caption](https://arxiv.org/html/2607.09692v1/x22.png)Figure 19:We apply reference\-based MIA toGPT\-OSS\-120B, usingGPT\-OSS\-20Bas the reference model\.![Refer to caption](https://arxiv.org/html/2607.09692v1/x23.png)Figure 20:We apply reference\-based MIA toGPT\-OSS\-120BandGPT\-OSS\-20B, usingGPT\-2 XLas the reference model and OMI as probing data\.![Refer to caption](https://arxiv.org/html/2607.09692v1/x24.png)Figure 21:Results for the GPT\-OSS family\.We evaluate these models with both reference\-based MIA \(left\) and our ASCII–Unicode diagnostic \(right\)\.Table 16:Statistical significance of theo1Unicode–ASCII diagnostic\.The within\-model gapδUni−ASCII\\delta\_\{\\text\{Uni\}\-\\text\{ASCII\}\}tested one\-sided against0\(Wilcoxon where the differences are non\-normal,tt\-test otherwise\); ✓ marksp<0\.05p<0\.05\. A significant gap means the separation is resolvable rather than sampling noise—suggestive ofo1\-style signal, not a confirmed distillation call\.