idSCD: Identifying Training Datasets through Semantic Correlation Descriptors

# idSCD: Identifying Training Datasets through Semantic Correlation Descriptors
Source: [https://arxiv.org/html/2605.30462](https://arxiv.org/html/2605.30462)
Andrada Gobeajă POLITEHNICA University of Bucharest andrada\.gobeaja@stud\.acs\.upb\.ro &Ionuț Hodoroagă11footnotemark:1 POLITEHNICA University of Bucharest ionut\.hodoroaga@stud\.acs\.upb\.ro Elena Burceanu POLITEHNICA University of Bucharest Bitdefender, Romania eburceanu@bitdefender\.com &Marius Leordeanu POLITEHNICA University of Bucharest Institute of Mathematics of the Romanian Academy marius\.leordeanu@upb\.ro

###### Abstract

Can a dataset be recognized from the spurious correlations it induces during training? We argue that datasets leave dataset\-specific traces in a model’s learned semantic correlation structure: incidental regularities that are predictive within a dataset, but not causal for the underlying task, can be internalized during training\. We use this insight to study dataset\-level membership inference, moving beyond existing methods that rely on behavioral or distributional evidence such as confidence scores, losses, margins, generated samples, or query responses\. We introduce a white\-box semantic fingerprinting approach based on semantic correlation descriptors \(SCDs\), which capture the semantic correlation structure learned by a model and make it comparable across dataset mixtures\. In a controlled leave\-one\-dataset\-out diagnostic, SCDs recover dataset\-specific changes and perfectly separate matching from non\-matching dataset pairs\. We then propose a practical SCD\-based membership score that tests whether a target dataset is part of a model’s training mixture using only the model’s SCD and the target dataset’s standalone SCD, without requiring leave\-one\-dataset\-out models\. Across three diverse experimental settings, with dataset groups for natural language inference, emotion classification, and medical text classification, we test both the advantages and limitations of SCD\-based membership inference with different degrees of semantic separation and keyword support between dataset splits\. On average, the classifieri​dSCDid\_\{\\mathrm\{SCD\}\}based on this score achieves the highest performance and the lowest std, outperforming black\-box baselines RMIA, Attack\-P, and LiRA, as well as the white\-box SIF baseline\. These results show that dataset membership can be traced through internal semantic correlations, with the largest relative gain exceeding60%60\\%in ROC\-AUC when dataset groups expose distinct semantic particularities\.

## 1Introduction

Determining whether a dataset was used to train a model is increasingly important for model auditing\. Modern models are trained on mixtures of public, private, licensed, synthetic, and domain\-specific data, often with limited visibility into which sources were used\. Dataset usage is therefore not merely a training detail: it affects privacy, licensing, benchmark contamination, reproducibility, and accountability\. Prior work has studied related questions through membership inference, dataset\-level membership inference, dataset inference, data provenance, and training\-data attribution\(Carlini et al\.,[2022](https://arxiv.org/html/2605.30462#bib.bib5); Maini et al\.,[2024](https://arxiv.org/html/2605.30462#bib.bib25); Sun et al\.,[2025](https://arxiv.org/html/2605.30462#bib.bib45)\)\. Most existing evidence for dataset membership is output\-driven\. Methods typically rely on confidence scores, losses, prediction margins, generated samples, query responses, or influence estimates\(Carlini et al\.,[2022](https://arxiv.org/html/2605.30462#bib.bib5); Ye et al\.,[2022](https://arxiv.org/html/2605.30462#bib.bib49); Cohen and Giryes,[2022](https://arxiv.org/html/2605.30462#bib.bib8)\)\. These signals can be effective, but they primarily ask whether a model behaves as if it has seen a dataset\. This makes them vulnerable to cases where different datasets induce similar output behavior, or where membership evidence is expressed less in predictions than in the internal structure learned by the model\. We ask a complementary question: can a dataset be recognized from the semantic correlations it induces during training?

Our starting point is that datasets differ in incidental regularities arising from their construction, domain, annotation protocol, and label expression patterns\. These regularities may be predictive within a dataset without being causal for the underlying task\. When a model is trained on a mixture of datasets, such regularities can be internalized as spurious semantic correlations and persist in the trained model\. Thus, even datasets mapped to the same label space may leave different fingerprints in how semantic features align with classes\. This suggests a different route to dataset\-level membership inference: instead of probing the model only from the outside, we audit the semantic association structure learned inside the model\. We introduce Semantic Correlation Descriptors \(SCDs\), which summarize this structure as dataset\-level fingerprints and allow models trained on different mixtures to be compared in a shared correlation space\. The goal is not to explain individual keywords, but to test whether the global semantic fingerprint of a target dataset is present in a model under inspection\.

First, in a controlled leave\-one\-dataset\-out diagnostic, SCDs recover dataset\-specific changes and perfectly separate matching from non\-matching dataset pairs, showing that the signal is not a generic training artifact\. Second, in a more realistic membership setting, we use an SCD\-based score that requires only the inspected model and a standalone reference descriptor for the target dataset, avoiding leave\-one\-dataset\-out models\. Across natural language inference, emotion classification, and medical text classification, this classifier achieves the best average performance and lowest standard deviation against recent strong baselines, with the largest gains when dataset splits expose distinct semantic particularities\.

Ourmain contributionsare the following:

1\) A semantic view of dataset\-level membership inference\.Existing dataset\-level membership methods primarily rely on behavioral or distributional evidence, such as confidence, loss, prediction margins, generated samples, or query responses\. We instead formulate dataset membership as a semantic tracing problem: can we recover the presence of a target dataset from the high\-level keyword\-class correlations it induces in the trained model?

2\) Descriptor for dataset fingerprinting\.We introduce a Semantic Correlation Descriptor \(SCDs\), denoted𝐝SCD\\mathbf\{d\}\_\{\\mathrm\{SCD\}\}, a compact representation for keyword\-class semantic correlation maps\. Through a shared keyword vocabulary and zero\-fill alignment, SCDs place models trained on different dataset mixtures in the same correlation space, enabling direct comparison of their learned association structure\. In a controlled leave\-one\-dataset\-out analysis, this representation perfectly separates matching from non\-matching dataset pairs\.

3\) A practical SCD\-based membership score\.We propose a membership score that tests whether a target datasetDiD\_\{i\}is part of the training mixture of a model under inspection, using only the model’s SCD and the standalone SCD ofDiD\_\{i\}\. Unlike the controlled analysis, it does not require leave\-one\-dataset\-out models\. Across three diverse experimental settings for natural language inference, emotion classification, and medical text classification, we test both the advantages and limitations of SCD\-based membership inference with different degrees of semantic separation and keyword support between dataset splits\. On average, the classifieri​dSCDid\_\{\\mathrm\{SCD\}\}induced by this score achieves the highest performance and the lowest standard deviation, outperforming recent strong baselines: the black\-box methods RMIA, Attack\-P, and LiRA, and the white\-box SIF baseline\.

## 2Related work

#### Relation to Spurious Correlations\.

Spurious correlations arise when models rely on superficial, non\-causal cues, such as backgrounds, textures, or secondary objects in vision tasks\(Geirhos et al\.,[2020](https://arxiv.org/html/2605.30462#bib.bib15); Wang et al\.,[2021](https://arxiv.org/html/2605.30462#bib.bib47); Yang et al\.,[2022](https://arxiv.org/html/2605.30462#bib.bib48); Lin et al\.,[2024](https://arxiv.org/html/2605.30462#bib.bib23)\)and keywords in language tasks\(Gururangan et al\.,[2018](https://arxiv.org/html/2605.30462#bib.bib16); McCoy et al\.,[2019](https://arxiv.org/html/2605.30462#bib.bib26); Niven and Kao,[2019](https://arxiv.org/html/2605.30462#bib.bib29); Du et al\.,[2023](https://arxiv.org/html/2605.30462#bib.bib11)\), instead of causal concepts\. These shortcuts encode dataset\-specific patterns that generalize poorly beyond the training distribution\(Geirhos et al\.,[2020](https://arxiv.org/html/2605.30462#bib.bib15); Arjovsky et al\.,[2020](https://arxiv.org/html/2605.30462#bib.bib1)\), often yielding higher confidence on samples containing familiar cues and greater uncertainty on out\-of\-distribution or clean test samples\(Izmailov et al\.,[2022](https://arxiv.org/html/2605.30462#bib.bib19); Ye et al\.,[2024](https://arxiv.org/html/2605.30462#bib.bib50)\)\. This confidence gap can be exploited by membership inference attacks\(Nasr et al\.,[2018](https://arxiv.org/html/2605.30462#bib.bib27); Choquette\-Choo et al\.,[2021](https://arxiv.org/html/2605.30462#bib.bib7)\)\. We show that such shortcut patterns can also be recovered at the dataset level as semantic correlation fingerprints, enabling dataset\-level membership inference\.

#### Relation to Membership Inference Attacks\.

Membership inference attacks \(MIAs\) determine whether a sample was used to train a machine learning model\(Shokri et al\.,[2017](https://arxiv.org/html/2605.30462#bib.bib42); Carlini et al\.,[2022](https://arxiv.org/html/2605.30462#bib.bib5)\)\. They exploit systematic differences between model behavior on training and unseen examples, and are typically classified by attacker access\. White\-box MIAs use internal signals such as activations, gradients, losses, confidence scores, parameters, or updates\(Papernot et al\.,[2018](https://arxiv.org/html/2605.30462#bib.bib32); Song and Mittal,[2020](https://arxiv.org/html/2605.30462#bib.bib43); Rigaki and Garcia,[2023](https://arxiv.org/html/2605.30462#bib.bib38)\), including gradient/activation attacks\(Nasr et al\.,[2019](https://arxiv.org/html/2605.30462#bib.bib28)\), Bayes\-optimal loss tests\(Sablayrolles et al\.,[2019](https://arxiv.org/html/2605.30462#bib.bib39)\), memorized feature associations\(Leino and Fredrikson,[2020](https://arxiv.org/html/2605.30462#bib.bib21)\), and self\-influence scores\(Cohen and Giryes,[2022](https://arxiv.org/html/2605.30462#bib.bib8)\)\. Black\-box MIAs rely only on query outputs such as labels, confidence scores, logits, entropy, or losses\(Li and Zhang,[2021](https://arxiv.org/html/2605.30462#bib.bib22); Liu et al\.,[2022](https://arxiv.org/html/2605.30462#bib.bib24); Carlini et al\.,[2022](https://arxiv.org/html/2605.30462#bib.bib5); Zarifzadeh et al\.,[2024](https://arxiv.org/html/2605.30462#bib.bib52)\), covering shadow\-model attacks\(Shokri et al\.,[2017](https://arxiv.org/html/2605.30462#bib.bib42)\), threshold attacks\(Yeom et al\.,[2018](https://arxiv.org/html/2605.30462#bib.bib51); Salem et al\.,[2018](https://arxiv.org/html/2605.30462#bib.bib40)\), label\-only attacks\(Choquette\-Choo et al\.,[2021](https://arxiv.org/html/2605.30462#bib.bib7); Peng et al\.,[2024](https://arxiv.org/html/2605.30462#bib.bib34)\), and calibrated likelihood\-ratio attacks\(Carlini et al\.,[2022](https://arxiv.org/html/2605.30462#bib.bib5); Bertran et al\.,[2023](https://arxiv.org/html/2605.30462#bib.bib2)\)\. Prior MIAs decide membership for individual examples using per\-example signals\. In contrast, we study dataset\-level membership: given a target datasetDiD\_\{i\}, we ask whetherDiD\_\{i\}appeared during training, using the semantic correlation structure induced by the dataset as membership signal\.

#### Relation to Training Data Attribution\.

Training data attribution methods seek to identify which training data most strongly affects a model’s output\. Prior work has approached this through influence functions, which approximate the effect of upweighting or removing training points\(Koh and Liang,[2020](https://arxiv.org/html/2605.30462#bib.bib20)\), datamodels, which learn counterfactual predictors from subset membership\(Ilyas et al\.,[2022](https://arxiv.org/html/2605.30462#bib.bib18)\), and scalable variants such as TRAK\(Park et al\.,[2023](https://arxiv.org/html/2605.30462#bib.bib33)\), In\-Run Data Shapley\(Wang et al\.,[2025](https://arxiv.org/html/2605.30462#bib.bib46)\), TrackStar\(Chang et al\.,[2024](https://arxiv.org/html/2605.30462#bib.bib6)\), and AirRep\(Sun et al\.,[2025](https://arxiv.org/html/2605.30462#bib.bib45)\), which reduce the cost of attribution through kernel approximations, single\-run contribution estimates, LLM\-scale gradient influence, or learned attribution representations\(Chang et al\.,[2024](https://arxiv.org/html/2605.30462#bib.bib6)\)\.Our method instead targets dataset\-level membership inference, asking whether an entire dataset was part of a model’s training mixture\. We construct a Semantic Correlation Descriptor \(SCDs\) from keyword\-class correlation maps and interpret dataset\-induced spurious correlations as dataset\-specific signatures\.

![Refer to caption](https://arxiv.org/html/2605.30462v1/main_figure_SCD.png)Figure 1:Dataset membership through semantic correlation fingerprints\. The pipeline evaluates whether dataset\-specific regularities leave recoverable traces in the semantic correlation structure of trained models\. We extract semantic correlations over a keyword vocabulary shared within each dataset group, aggregate them into Semantic Correlation Descriptors \(SCDs\), and use the SCD classifier over the membership score to test whether a target dataset was part of the training mixture\.

## 3Our approach

Datasets differ in the incidental regularities that arise from their specific construction, domain, annotation protocol, and label expression patterns\. During training, these regularities can be internalized by the model as spurious correlations: associations between surface keywords and classes that are predictive within a dataset, but are not causal features of the underlying task\. When a model is trained on a mixture of datasets, such dataset\-specific correlations may be absorbed into its learned decision structure and persist after training\. We ask whether the presence of a target dataset can be recovered from the spurious correlations it induces during training\. We study this question in a controlled setting with a group of datasetsD1,…,DND\_\{1\},\\ldots,D\_\{N\}, each mapped to a shared label space𝒴\\mathcal\{Y\}, and a fixed model architecture\. Given a target datasetDiD\_\{i\}, our goal is to infer whetherDiD\_\{i\}was present in the training mixture\. To formalize this, let𝒰\\mathcal\{U\}denote the global keyword vocabulary associated with the dataset collection\. For each trained modelmm, we construct a keyword\-class correlation matrix:

𝐂\(m\)∈ℝ\|𝒰\|×\|𝒴\|,\\mathbf\{C\}^\{\(m\)\}\\in\\mathbb\{R\}^\{\|\\mathcal\{U\}\|\\times\|\\mathcal\{Y\}\|\},where each entry𝐂u,y\(m\)\\mathbf\{C\}^\{\(m\)\}\_\{u,y\}quantifies the strength of the correlation between keywordu∈𝒰u\\in\\mathcal\{U\}and labely∈𝒴y\\in\\mathcal\{Y\}in modelmm\. Rather than interpreting individual keyword\-class correlations, we treat the full correlation map as a dataset\-level fingerprint and use it to infer whetherDiD\_\{i\}was present in the training mixture through its dataset\-specific spurious correlation patterns\.

### 3\.1From semantic correlations to dataset fingerprints

For each dataset group, we train three model variants designed to make dataset\-specific contributions observable\. For each target datasetDiD\_\{i\}, we first train a single\-dataset modelmim\_\{i\}using onlyDiD\_\{i\}\. We also train a full modelmallm\_\{\\mathrm\{all\}\}on the union of all datasets,Dall=⋃i=1NDiD\_\{\\mathrm\{all\}\}=\\bigcup\_\{i=1\}^\{N\}D\_\{i\}, and a leave\-one\-dataset\-out modelm−im\_\{\-i\}on the union of all datasets exceptDiD\_\{i\},D−i=⋃j≠iDjD\_\{\-i\}=\\bigcup\_\{j\\neq i\}D\_\{j\}\.

#### Semantic correlation maps\.

Our approach requires a scoring function that estimates the degree of keyword\-class correlation across trained models\. In our implementation, we obtain the correlation matrices𝐂\\mathbf\{C\}using BEE\(Paduraru et al\.,[2026](https://arxiv.org/html/2605.30462#bib.bib31)\), which extracts keyword\-class correlations by comparing keyword embeddings and class weight vectors in a shared embedding space\. In BEE, each trained model is converted into a keyword\-class correlation matrix\. For a modelmm, each keywordu∈𝒰u\\in\\mathcal\{U\}is embedded with the model’s last\-token hidden state, giving a keyword embedding𝐞u\(m\)\\mathbf\{e\}\_\{u\}^\{\(m\)\}\. Let𝐰y\(m\)\\mathbf\{w\}\_\{y\}^\{\(m\)\}be the classifier’s vector for classyy, extracted after training, from the sequence\-classification head\. The keyword\-class alignment is computed by cosine similarity,simm​\(u,y\)=cos⁡\(𝐞u\(m\),𝐰y\(m\)\)\\mathrm\{sim\}\_\{m\}\(u,y\)=\\cos\\\!\\left\(\\mathbf\{e\}\_\{u\}^\{\(m\)\},\\mathbf\{w\}\_\{y\}^\{\(m\)\}\\right\)\. The cosine alignment is then refined into the BEE scoring function for keyword\-class correlations \- which yields entries of the correlation matrix where rows correspond to keywords and columns to labels:

𝐂u,y\(m\)=BEEm​\(u,y\)=simm​\(u,y\)−miny′∈𝒴⁡simm​\(u,y′\)\.\\mathbf\{C\}^\{\(m\)\}\_\{u,y\}=\\mathrm\{BEE\}\_\{m\}\(u,y\)=\\mathrm\{sim\}\_\{m\}\(u,y\)\-\\min\_\{y^\{\\prime\}\\in\\mathcal\{Y\}\}\\mathrm\{sim\}\_\{m\}\(u,y^\{\\prime\}\)\.\(1\)

#### Zero\-fill alignment\.

Keyword extraction tools generally produce keyword sets of different sizes across datasets\. As a result, the number of rows of the corresponding correlation matrices𝐂\(mi\)\\mathbf\{C\}^\{\(m\_\{i\}\)\}may differ\. To bring these matrices to the same size we use a shared global keyword vocabulary𝒰\\mathcal\{U\}within each dataset group\. For each modelmim\_\{i\}, we construct𝐂\(mi\)\\mathbf\{C\}^\{\(m\_\{i\}\)\}over the same vocabulary𝒰\\mathcal\{U\}\. If keyworduuis present in the keyword set extracted for modelmim\_\{i\}, its BEE scores are inserted into the corresponding row of𝐂\(mi\)\\mathbf\{C\}^\{\(m\_\{i\}\)\}\. Ifuuis absent, that row is set to zero:

𝐂u,y\(mi\)=\{BEEmi​\(u,y\),if keyword​u​is present for model​mi,0,otherwise\.\\mathbf\{C\}^\{\(m\_\{i\}\)\}\_\{u,y\}=\\begin\{cases\}\\mathrm\{BEE\}\_\{m\_\{i\}\}\(u,y\),&\\text\{if keyword \}u\\text\{ is present for model \}m\_\{i\},\\\\\[4\.0pt\] 0,&\\text\{otherwise\.\}\\end\{cases\}\(2\)
This zero\-fill alignment yields correlation matrices with a common shape,\|𝒰\|×\|𝒴\|\|\\mathcal\{U\}\|\\times\|\\mathcal\{Y\}\|, enabling direct comparison of trained models from the perspective of their learned keyword\-class correlations\.

#### Semantic Correlation Descriptor \(SCD\)\.

For each datasetDiD\_\{i\}, we define its correlation descriptor as the vectorized correlation map of the corresponding single\-dataset modelmim\_\{i\}:dSCD\(i\)=vec​\(𝐂\(mi\)\)d\_\{\\mathrm\{SCD\}\}^\{\(i\)\}=\\mathrm\{vec\}\\\!\\left\(\\mathbf\{C\}^\{\(m\_\{i\}\)\}\\right\)By construction,dSCD\(i\)d\_\{\\mathrm\{SCD\}\}^\{\(i\)\}captures the keyword\-class correlation pattern learned fromDiD\_\{i\}in isolation\. This pattern includes the spurious correlations induced by the dataset, and serves as a dataset\-specific reference for comparing learned correlation structures\.

### 3\.2SCD sensitivity analysis in a controlled setting

Before defining the final membership score, we first test whether SCDs capture dataset\-specific correlation patterns in a controlled setting\. For each target datasetDiD\_\{i\}, we compare the full modelmallm\_\{\\mathrm\{all\}\}with the leave\-one\-dataset\-out modelm−im\_\{\-i\}\. The goal of this analysis is not to define the final membership inference protocol, but to verify whether the change between these two models aligns with the standalone descriptor ofDiD\_\{i\}, rather than with the descriptors of other datasets\. We estimate the dataset\-specific change associated withDiD\_\{i\}as

Δi=dSCD\(all\)−dSCD\(−i\)\.\\Delta\_\{i\}=d\_\{\\mathrm\{SCD\}\}^\{\(\\mathrm\{all\}\)\}\-d\_\{\\mathrm\{SCD\}\}^\{\(\-i\)\}\.\(3\)Intuitively,Δi\\Delta\_\{i\}captures the change in the learned keyword\-class correlation structure associated withDiD\_\{i\}\. IfDiD\_\{i\}induces a distinctive correlation pattern, then this estimated change should be more similar to the standalone descriptor ofDiD\_\{i\}than to the descriptors of other datasets\. We measure this alignment using Pearson correlation\. For each target datasetDiD\_\{i\}and each candidate datasetDjD\_\{j\}, we define

sΔ​\(i,j\)=Pearson​\(Δi,dSCD\(j\)\)=Pearson​\(dSCD\(all\)−dSCD\(−i\),dSCD\(j\)\)\.s\_\{\\Delta\}\(i,j\)=\\mathrm\{Pearson\}\\left\(\\Delta\_\{i\},d\_\{\\mathrm\{SCD\}\}^\{\(j\)\}\\right\)=\\mathrm\{Pearson\}\\left\(d\_\{\\mathrm\{SCD\}\}^\{\(\\mathrm\{all\}\)\}\-d\_\{\\mathrm\{SCD\}\}^\{\(\-i\)\},d\_\{\\mathrm\{SCD\}\}^\{\(j\)\}\\right\)\.\(4\)
This score induces a binary test over dataset pairs\(i,j\)\(i,j\): pairs withj=ij=icorrespond to matching descriptors, while pairs withj≠ij\\neq icorrespond to non\-matching descriptors\. A simple threshold classifier can therefore be defined asz^i,j=𝟏​\[sΔ​\(i,j\)\>τ\]\\hat\{z\}\_\{i,j\}=\\mathbf\{1\}\\left\[s\_\{\\Delta\}\(i,j\)\>\\tau\\right\], wherez^i,j=1\\hat\{z\}\_\{i,j\}=1predicts thatΔi\\Delta\_\{i\}matches the descriptor ofDjD\_\{j\}, andz^i,j=0\\hat\{z\}\_\{i,j\}=0otherwise\.

The resulting matrixSΔ∈ℝN×NS\_\{\\Delta\}\\in\\mathbb\{R\}^\{N\\times N\}measures how well each estimated change matches each standalone dataset descriptor\. Diagonal dominance,sΔ​\(i,i\)\>sΔ​\(i,j\)s\_\{\\Delta\}\(i,i\)\>s\_\{\\Delta\}\(i,j\)for allj≠ij\\neq i, indicates that the change associated withDiD\_\{i\}is best explained by the fingerprint ofDiD\_\{i\}itself\. In Sec\.[4](https://arxiv.org/html/2605.30462#S4), we use this controlled analysis to validate the dataset\-specificity of SCDs: the threshold classifier based onsΔs\_\{\\Delta\}perfectly separates matching from non\-matching pairs, achieving a ROC\-AUC of100%100\\%in Fig\.[2](https://arxiv.org/html/2605.30462#S4.F2)\. This controlled analysis is not intended as the final membership inference protocol, since it assumes access to bothmallm\_\{\\mathrm\{all\}\}andm−im\_\{\-i\}\. Rather, it serves as a diagnostic test of whether SCDs respond to dataset\-specific changes in the learned correlation structure\. In the next section, we consider a more realistic setting and define a membership score that does not require leave\-one\-dataset\-out models\.

### 3\.3SCD\-based dataset membership inference

Instead of requiring paired models trained with and without a target datasetDiD\_\{i\}, we assume access only to a trained modelmanym\_\{\\mathrm\{any\}\}and to the single\-dataset reference descriptor ofDiD\_\{i\}\. The goal is to decide whetherDiD\_\{i\}was included in the training data ofmanym\_\{\\mathrm\{any\}\}\.

The key idea is that, ifmanym\_\{\\mathrm\{any\}\}was trained onDiD\_\{i\}, then the dataset\-specific correlation pattern captured by the descriptor ofDiD\_\{i\}should also be visible in the correlation descriptor ofmanym\_\{\\mathrm\{any\}\}\. We therefore compare the two descriptors directly using Pearson correlation:

sSCD​\(i\)=Pearson​\(dSCD\(any\),dSCD\(i\)\)\.s\_\{\\mathrm\{SCD\}\}\(i\)=\\mathrm\{Pearson\}\\left\(d\_\{\\mathrm\{SCD\}\}^\{\(\\mathrm\{any\}\)\},d\_\{\\mathrm\{SCD\}\}^\{\(i\)\}\\right\)\.\(5\)
Higher values ofsSCD​\(i\)s\_\{\\mathrm\{SCD\}\}\(i\)indicate stronger evidence that the correlation pattern associated withDiD\_\{i\}is present inmanym\_\{\\mathrm\{any\}\}\. This score essentially measures how strongly the descriptor of the target dataset matches the descriptor of the model under inspection\. UnlikesΔs\_\{\\Delta\}, it does not require training or accessing models explicitly constructed withDiD\_\{i\}and withoutDiD\_\{i\}, making it suitable for real\-life membership inference settings\. We usesSCDs\_\{\\mathrm\{SCD\}\}to define a threshold\-based membership classifier:

i​d^SCD\(i\)=𝟏​\[sSCD​\(i\)\>τ\],\\hat\{id\}\_\{\\mathrm\{SCD\}\}^\{\(i\)\}=\\mathbf\{1\}\\left\[s\_\{\\mathrm\{SCD\}\}\(i\)\>\\tau\\right\],\(6\)wherei​d^SCD\(i\)=1\\hat\{id\}\_\{\\mathrm\{SCD\}\}^\{\(i\)\}=1predicts thatDiD\_\{i\}is included in the training data ofmanym\_\{\\mathrm\{any\}\}, andi​d^SCD\(i\)=0\\hat\{id\}\_\{\\mathrm\{SCD\}\}^\{\(i\)\}=0otherwise\. In Sec\.[4](https://arxiv.org/html/2605.30462#S4), we evaluate this classifier by measuring the ROC\-AUC obtained fromsSCDs\_\{\\mathrm\{SCD\}\}in Fig[3](https://arxiv.org/html/2605.30462#S4.F3)\.

## 4Experiments

#### Dataset groups\.

We evaluate dataset\-level attribution and membership inference on three dataset groups: native language identification \(NLI\), emotion classification, and medical section classification\. The emotion group contains four heterogeneous datasets: EmoLit\(Rei and Mladenić,[2023](https://arxiv.org/html/2605.30462#bib.bib37)\), EmotionDataset20\(Pulle,[2025](https://arxiv.org/html/2605.30462#bib.bib35)\), GoEmotions\(Demszky et al\.,[2020](https://arxiv.org/html/2605.30462#bib.bib9)\), and a merged XED\(Öhman et al\.,[2020](https://arxiv.org/html/2605.30462#bib.bib30)\)/SMED\(Ganguly,[2023](https://arxiv.org/html/2605.30462#bib.bib13)\)corpus\. The NLI group contains three datasets of English text written by non\-native speakers: Reddit\-L2, EFCAMDAT, and iTalki\. The medical section classification group is derived from the PubMed 200k RCT corpus\(Dernoncourt and Lee,[2017](https://arxiv.org/html/2605.30462#bib.bib10)\)\. In every group, all datasets are mapped to a shared set of common classes so that keyword\-class association maps are comparable across models trained on different dataset subsets\.

#### Emotion dataset group\.

The emotion group combines four sources that differ in genre, granularity, and annotation protocol\.EmoLitis treated as multi\-label over fine\-grained emotion dimensions: after deduplicating sentences, we assign each example the emotion dimension with the largest real\-valued score and then map that winner to a coarse label using a fixed many\-to\-one scheme\.GoEmotionsmerges the train, development, and test splits; we drop duplicate comments, and for multi\-labeled rows we take the*first*emotion id in the comma\-separated list as the provisional fine\-grained tag before mapping to the same coarse inventory\.EmotionDataset20uses the CSV text field and single per\-sentence emotion string; labels are normalized \(lowercased\) and mapped with the same coarse taxonomy\.XED\(English\) supplies integer ids that may enumerate several emotions per line\. We expand these into one row per id, map ids to string emotion names, and concatenate the result withSMED, after normalizing SMED’s string labels \(strip, title case\)\. The XED/SMED union is deduplicated on raw text\. All four mapped corpora are concatenated and deduplicated again ontext\(first occurrence kept\) to form a single pooled emotion dataset, while leave\-one\-corpus\-out training files are built by concatenating any three of the four mapped sources with the same text\-level deduplication \(see App\.[A\.2](https://arxiv.org/html/2605.30462#A1.SS2)for details\)\. Throughout, fine\-grained original labels are collapsed into nine shared classes:*Angry*,*Anticipation*,*Disgust*,*Fear*,*Happy*,*Neutral*,*Sad*,*Surprise*, and*Trust*, so that keyword\-class association rules remain comparable across models trained on different subsets\.

#### NLI datasets\.

The NLI evaluation relies on three foundational datasets: Reddit\-L2\(Rabinovich et al\.,[2018](https://arxiv.org/html/2605.30462#bib.bib36)\), EFCAMDAT\(Geertzen et al\.,[2013](https://arxiv.org/html/2605.30462#bib.bib14)\), and iTalki\(Hudson and Jaf,[2018](https://arxiv.org/html/2605.30462#bib.bib17)\)\. To avoid trivial membership inference signals caused by unique labels, we filter the corpora to retain only the five first\-language \(L1\) classes shared by all three sources: French, German, Italian, Spanish, and Turkish\. To ensure the models are not biased toward the larger corpora, we use stratified subsampling to Reddit\-L2 and EFCAMDAT, reducing them to the size of the smallest dataset \(iTalki\), as detailed in App\.[A\.2](https://arxiv.org/html/2605.30462#A1.SS2), while preserving original class distributions\. Due to its prohibitive size, Reddit\-L2 is subsampled by a chunked, two\-pass approach and EFCAMDAT is processed in\-memory\.

#### Medical datasets\.

Unlike the emotion and NLI groups, which inherently consist of multiple distinct datasets, the medical dataset group is a five\-way split created from PubMed 200k RCT, a single large biomedical corpus containing over 2 million sentences\. Following prior work on clustering\-based splits\(Firca et al\.,[2025](https://arxiv.org/html/2605.30462#bib.bib12)\), we partition it into semantically distinct topic clusters expected to induce clearer dataset fingerprints\. First, to accommodate the computational footprint of the 14\-million parameter LLM, Pythia\(Biderman et al\.,[2023](https://arxiv.org/html/2605.30462#bib.bib3)\), used in our experiments, we extract a 12% stratified subsample of the original dataset\. This subsample preserves the original class distribution of the rhetorical roles \(i\.e\., BACKGROUND, METHODS, RESULTS, CONCLUSIONS, OBJECTIVE\)\. To ensure the data is split based on overarching medical topics rather than sentence\-level syntax, we aggregate the individual sentences using theirabstract\_idto reconstruct the full text of each abstract\. We apply TF\-IDF\(Spärck Jones,[1972](https://arxiv.org/html/2605.30462#bib.bib44)\)\(Term Frequency\-Inverse Document Frequency\) to these reconstructed abstracts to extract features, removing standard English stop words and limiting the vocabulary to the top 10,000 terms\. We then apply the MiniBatch K\-Means\(Sculley,[2010](https://arxiv.org/html/2605.30462#bib.bib41)\)algorithm to the resulting TF\-IDF matrix to group the abstracts into five distinct topic clusters\. Crucially, we map these cluster assignments back to the original sentence\-level data\. This ensures that all sentences belonging to a single abstract are kept together in the same cluster, strictly preventing data leakage across different splits\.

#### Keyword extraction\.

Keywords are extracted independently for each trained\-model configuration using YAKE\(Campos et al\.,[2020](https://arxiv.org/html/2605.30462#bib.bib4)\)\. We use YAKE with maximum n\-gram lengthn=3n=3,dedupLim=0\.9, and 5000 keywords per model configuration, the resulting keyword pool containing unigrams, bigrams, and trigrams\. We filter class\-name keywords and empty or invalid tokens\. For each dataset group, we construct a global keyword pool𝒰\\mathcal\{U\}as the union of all extracted keywords across the full, leave\-one\-out, and single\-dataset models in that group\.

#### Implementation details\.

We use the Pythia\-14M architecture \(EleutherAI/pythia\-14m\) as the base model\. Each model is first finetuned as a causal language model and then used to initialize a sequence\-classification model\. The classifier uses theGPTNeoXForSequenceClassificationhead, exposed asmodel\.score; the class directions used by BEE are the rows ofmodel\.score\.weight\. Both stages are trained for three epochs with learning rate2×10−52\\times 10^\{\-5\}, weight decay 0\.01, epoch\-level evaluation and checkpointing, and mixed precision when CUDA is available\. All preprocessing, sampling, training, and evaluation use fixed random seed4242\. For each dataset group, we train individual, leave\-one\-dataset\-out, and full\-corpus models, generating 27 training configurations in total, as reported in App\.[A\.2](https://arxiv.org/html/2605.30462#A1.SS2)\.

#### Computational costs\.

Model training was run on NVIDIA A100 GPUs with 40GB VRAM on Google Colab, while dataset processing and keyword extraction used high\-RAM CPU workers with 40GB RAM\. Reproducing the main experiments requires approximately 42 GPU\-hours: 19 hours for the emotion group, 9 hours for NLI, and 14 hours for the medical group\. Reproducing the baseline comparisons requires an additional 30 GPU\-hours\.

#### Ablations and design choices\.

We explored several variants along the pipeline before settling on the final setup\. At the preprocessing level, we varied YAKE keyword extraction by changing thenn\-gram length \(n=1,5¯n=\\overline\{1,5\}\), the number of extracted keywords per dataset \(200,500,1000,2000,5000200,500,1000,2000,5000\), and class\-conditioned extraction strategies such as selecting the top10,100¯\\overline\{10,100\}keywords per class\. We also tested alternative medical split constructions, including PCA followed by K\-Means, before adopting TF\-IDF with K\-Means, and initially treated XED and SMED as separate emotion sources before merging them into a single dataset\. At the descriptor construction level, we evaluated variants without zero\-fill alignment, variants that did not merge matrices into a shared vocabulary space, and alternative BEE scoring choices, including softmax\-temperature variants withT=1,3¯T=\\overline\{1,3\}\. At the scoring level, we tested cosine similarity instead of Pearson correlation, as well as direct Pearson correlation over logits rather than BEE\-based descriptors\. These ablations guided the final design: shared\-vocabulary zero\-fill alignment, BEE\-based semantic correlation descriptors, and Pearson\-based SCD scoring\.

![Refer to caption](https://arxiv.org/html/2605.30462v1/x1.png)\(a\)Medical datasets group\.
![Refer to caption](https://arxiv.org/html/2605.30462v1/x2.png)\(b\)Emotion datasets group\.
![Refer to caption](https://arxiv.org/html/2605.30462v1/x3.png)\(c\)NLI datasets group\.

Figure 2:Dataset\-specificity of correlation descriptors in the controlled difference\-based setting\. Each heatmap shows theSΔS\_\{\\Delta\}matrix defined in Sec\.[3\.2](https://arxiv.org/html/2605.30462#S3.SS2), with rows corresponding to target datasets and columns corresponding to standalone dataset descriptors, defined in Eq\.[3](https://arxiv.org/html/2605.30462#S3.E3)\. Across different dataset groups, the strongest score in every row lies on the diagonal, showing that each estimated dataset\-specific change is best matched by the descriptor of the corresponding dataset\. Treating diagonal entries as positives and off\-diagonal entries as negatives yields perfect separation, with a ROC\-AUC of100%100\\%\.
### 4\.1Correlation Descriptors are dataset\-specific

We evaluate here whether the correlation descriptors defined in Sec\.[3\.1](https://arxiv.org/html/2605.30462#S3.SS1)are sensitive to dataset\-specific changes in the training mixture\. This experiment is designed as a controlled diagnostic for the intuition in Sec\.[3\.2](https://arxiv.org/html/2605.30462#S3.SS2): if a datasetDiD\_\{i\}contributes a characteristic correlation pattern, then the difference\-based descriptorΔi\\Delta\_\{i\}from Eq\.[3](https://arxiv.org/html/2605.30462#S3.E3)should align most strongly with the standalone descriptor ofDiD\_\{i\}, rather than with the descriptors of other datasets\.

#### Results\.

Fig\.[2](https://arxiv.org/html/2605.30462#S4.F2)shows the resultingSΔS\_\{\\Delta\}matrices for the NLI, emotion, and medical dataset groups\. The pattern is consistent across all groups: the largest score in each row appears on the diagonal\. This means that the estimated change associated with each target dataset is best matched by that dataset’s own correlation descriptor\. Off\-diagonal entries are substantially lower and often negative, indicating that these changes do not align with unrelated dataset descriptors\. We further quantify this separation by treating diagonal entries\(i=j\)\(i=j\)as positives and off\-diagonal entries\(i≠j\)\(i\\neq j\)as negatives\. A simple threshold classifier oversΔ​\(i,j\)s\_\{\\Delta\}\(i,j\)perfectly separates the two classes, achieving a ROC\-AUC of100%100\\%\. This confirms that, in this controlled setting, correlation descriptors are not merely capturing generic training artifacts\. They encode dataset\-specific correlation structure strongly enough to recover which dataset accounts for the observed change in the model\.

#### Takeaway\.

This experiment validates the core mechanism behind our approach: dataset\-specific correlation patterns are measurable, separable, and recoverable from trained models\. At the same time, this is a controlled diagnostic setting because it assumes access to the leave\-one\-dataset\-out modelm−im\_\{\-i\}\. In the next section, we move to a more realistic membership inference setting, where we do not rely on leave\-one\-dataset\-out models\.

### 4\.2SOTA Comparison for Membership Inference

We next compare the practical SCD\-based membership classifier from Sec\.[3\.3](https://arxiv.org/html/2605.30462#S3.SS3)against established membership inference baselines\. Unlike the controlled diagnostic, this setting does not rely on leave\-one\-dataset\-out differences\. The goal of this evaluation is to test whetheri​dSCDid\_\{\\mathrm\{SCD\}\}can distinguish models that were trained with a target dataset from models where that dataset was absent\. For each target dataset, positive examples are obtained from the full modelmallm\_\{\\mathrm\{all\}\}, while negative examples are obtained from the corresponding leave\-one\-dataset\-out modelm−im\_\{\-i\}\. We compute ROC\-AUC by varying the decision threshold ofi​dSCDid\_\{\\mathrm\{SCD\}\}, and compare against recent strong membership inference baselines: black\-box methods RMIA, Attack\-P, and LiRA, which rely on output\-derived membership signals such as loss, confidence and likelihood\-ratio, and the white\-box SIF baseline, which uses model\-internal information to compute self\-influence scores\.

#### Results\.

Fig\.[3](https://arxiv.org/html/2605.30462#S4.F3)shows that our SCD\-based classifier achieves the strongest performance on the medical and emotion groups, and remains competitive on NLI\. The largest gains appear on the medical group, wherei​dSCDid\_\{\\mathrm\{SCD\}\}improves over all baselines by more than60%60\\%\(see the discussion below for why this is the case\)\. On emotion,i​dSCDid\_\{\\mathrm\{SCD\}\}also achieves the best performance, improving over the strongest baseline by about8%8\\%and reaching a ROC\-AUC of0\.9800\.980\. On NLI,i​dSCDid\_\{\\mathrm\{SCD\}\}remains competitive, but is no longer the top method: it is about3%3\\%below the best baseline\. As discussed below, this gap is explained by the much smaller shared semantic support in the NLI setup, a condition that can be detected before applying our method and used to assess whether SCD\-based membership inference is suitable for a given dataset group\. Aggregated across groups,i​dSCDid\_\{\\mathrm\{SCD\}\}achieves the highest mean ROC\-AUC and the lowest std, indicating a stronger and more stable membership signal\. This suggests that SCDs are most effective when datasets induce distinct semantic correlation patterns, while baselines are more sensitive to non\-semantic dataset artifacts\.

#### Why medical benefits from SCDs\.

The strongest improvement \(\+60% in ROC\-AUC\) appears on the medical dataset group, and this is expected from the construction of the benchmark\. Unlike NLI and emotion, the medical group is built by splitting each class in PubMed 200k RCT into five topic clusters using high\-level semantic features\. This creates datasets that differ substantially in their semantic content while preserving the same rhetorical\-role label space\. Such a construction directly favors methods that use internal semantic correlations, because the resulting splits induce more separated feature\-class semantic association patterns\. In contrast, output\-based attacks may not fully capture these high\-level semantic differences if the model’s predictive behavior remains similar across clusters\. The medical result therefore highlights the main advantage of SCDs: when datasets differ semantically, their internal correlation fingerprints can be more discriminative than output behavior\.

#### Limited SCD suitability on NLI\.

The smaller margin on NLI is also informative\. Our method compares semantic correlations over a keyword vocabulary shared across datasets, so the effective decision signal depends on how much semantic overlap is exposed through shared keywords between the target dataset and the inspected model’s descriptor\. In the NLI group, the keyword overlap between two datasets is only 15%, while in the other groups is 20%\. This leaves the SCD classifier with less shared semantic evidence and explains why the SCD signal is less dominant on NLI: the method is designed to exploit correlations between shared keywords and classes, and its discriminative power decreases when the common keyword support is sparse\. See App\.[A\.3](https://arxiv.org/html/2605.30462#A1.SS3)for details\.

#### Takeaway\.

These results suggest that semantic correlation structure can substantially change how dataset\-level membership is detected: when datasets induce distinct semantic fingerprints, gains over output\-based and influence\-based baselines can be large\. Across our three evaluation groups, SCDs are strongest when datasets are semantically separable and share enough keyword support, while the baselines oscillate much more, suggesting higher sensitivity to non\-semantic dataset\-specific artifacts\. Importantly, shared\-keyword support can be checked a priori, providing a simple diagnostic for whether SCD\-based inference is sufficiently semantically informed for a given dataset group\.

![Refer to caption](https://arxiv.org/html/2605.30462v1/x4.png)\(a\)Medical datasets group\.
![Refer to caption](https://arxiv.org/html/2605.30462v1/x5.png)\(b\)Emotion group\.
![Refer to caption](https://arxiv.org/html/2605.30462v1/x6.png)\(c\)NLI datasets group\.
![Refer to caption](https://arxiv.org/html/2605.30462v1/x7.png)\(d\)Mean across groups\.

Figure 3:SOTA comparison of ROC curves for dataset membership inference\. \(a\-c\) show performance on each dataset group, while \(d\) summarizes the mean and standard deviation \(std\) across groups\. We compare against black\-box baselines RMIA, Attack\-P, and LiRA, and white\-box SIF baseline\. We achieve the top performance on the medical and emotion groups, are competitive on NLI, and obtain the highest average performance with lowest std\. In contrast, the baselines fluctuate more across groups, suggesting they are more sensitive to non\-semantic dataset\-specific artifacts\.

## 5Limitations

#### White\-box access\.

Our strongest instantiation of the method uses internal model information, including class weights and embedding\-space associations\. This is a limitation for fully closed models, where such quantities are not directly available\. However, this assumption matches many realistic auditing settings, including internal model audits, open\-weight models, model handoffs, and regulated provenance checks where the auditor can inspect model parameters\. Moreover, the framework is not intrinsically tied to this exact signal: the correlation descriptor can be instantiated with alternative scoring functions, including output\-based probes or descriptors extracted from distilled surrogate models\. These variants may provide weaker evidence, but they offer a path toward applying the same dataset\-level tracing principle beyond strict white\-box access\.

#### Dataset identity versus dataset similarity\.

Our method operates at the level of dataset descriptors, not individual training examples\. A strong match should therefore be interpreted as evidence that the model contains a dataset\-level correlation pattern associated withDiD\_\{i\}, not as a sample\-level proof that every example inDiD\_\{i\}was used during training\. This distinction matters when datasets share domain, style, label semantics, or annotation artifacts: a high score may reflect genuine use ofDiD\_\{i\}, but it may also arise from a closely related dataset with a similar correlation structure\. The leave\-one\-dataset\-out analysis tests whether the recovered contribution is better explained byDiD\_\{i\}than by other datasets, which reduces this ambiguity within the evaluated dataset groups\. Still, absolute dataset identity is difficult when two datasets are near duplicates or intentionally constructed to share the same artifacts\.

#### Dependence on shared keyword support\.

SCD\-based inference relies on comparing semantic correlations over keywords that are shared across datasets\. When the overlap between dataset\-specific keyword vocabularies is sparse, the descriptor comparison has less semantic evidence to exploit, and the membership signal can become weaker\. This limitation is partly diagnosable before running the method: low shared\-keyword overlap indicates that the dataset group may be less suitable for SCD\-based membership inference\. In our experiments, this effect is visible in the NLI group, where lower keyword overlap reduces the discriminative power of the SCD classifier\.

#### Class alignment \(evaluation framework limitation\)\.

Our evaluation requires datasets within a group to be mapped to a shared set of canonical labels before fingerprinting\. This is primarily a constraint of the current test framework, rather than a fundamental requirement that all datasets share identical labels\. In practice, many scenarios already involve groups of datasets for the same task, such as NLI, emotion classification, or medical text classification, where such mappings are meaningful\. We evaluate the method across three dataset groups to avoid relying on a single label taxonomy\. But the quality of canonicalization matters: ambiguous, hierarchical, or partially overlapping labels can blur correlation descriptors and make dataset\-specific patterns harder to separate\.

#### Scalability of controlled validation \(evaluation framework limitation\)\.

The leave\-one\-dataset\-out protocol used in our controlled sensitivity analysis requires training one model per removed dataset, which becomes expensive as the number of candidate datasets grows\. This cost is a limitation of the validation protocol, not of the final membership score itself\. The controlled setting is useful because it isolates the contribution of each dataset and tests whether correlation descriptors respond to dataset\-specific changes\. For larger collections, this ablation\-based analysis could be replaced or approximated by more efficient strategies, such as subset ablations, influence\-style approximations, low\-rank descriptor updates, or surrogate models\. Importantly, our more realistic membership scoring setting does not require training a full leave\-one\-dataset\-out suite\.

## 6Conclusion

We show that a dataset can be recognized from the semantic correlations it induces during training\. Our work reframes dataset\-level membership inference as a white\-box semantic fingerprinting problem: instead of relying on output behavior, we trace datasets through the internal correlation structure learned by the model\. We instantiate this idea with Semantic Correlation Descriptors \(SCDs\), show in a controlled leave\-one\-dataset\-out diagnostic that they recover dataset\-specific changes with perfect separation, and introduce a practical SCD\-based membership score that only requires the inspected model’s SCD and the target dataset’s standalone SCD\. Across NLI, emotion classification, and medical text classification, we evaluate SCDs under different levels of semantic separation and keyword support\. On average, the resultingi​dSCDid\_\{\\mathrm\{SCD\}\}classifier achieves the highest performance and the lowest std among recent strong baselines, including RMIA, Attack\-P, LiRA, and SIF\. The largest gains appear when dataset groups expose distinct semantic particularities, reaching up to\+60%\+60\\%ROC\-AUC\. These results establish internal semantic correlations as a strong and stable signal for dataset provenance, complementary to behavioral and influence\-based membership inference\.

## Acknowledgments and Disclosure of Funding

This work is supported in part by projects “Romanian Hub for Artificial Intelligence \- HRIA”, Smart Growth, Digitization and Financial Instruments Program, 2021\-2027 \(MySMIS no\. 334906\) and "European Lighthouse of AI for Sustainability \- ELIAS", Horizon Europe program \(Grant No\. 101120237\)\.

## References

- Arjovsky et al\. \(2020\)Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez\-Paz\.Invariant risk minimization, 2020\.URL[https://arxiv\.org/abs/1907\.02893](https://arxiv.org/abs/1907.02893)\.
- Bertran et al\. \(2023\)Martin Bertran, Shuai Tang, Michael Kearns, Jamie Morgenstern, Aaron Roth, and Zhiwei Steven Wu\.Scalable membership inference attacks via quantile regression, 2023\.URL[https://arxiv\.org/abs/2307\.03694](https://arxiv.org/abs/2307.03694)\.
- Biderman et al\. \(2023\)Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal\.Pythia: A suite for analyzing large language models across training and scaling, 2023\.URL[https://arxiv\.org/abs/2304\.01373](https://arxiv.org/abs/2304.01373)\.
- Campos et al\. \(2020\)Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Jatowt\.Yake\! keyword extraction from single documents using multiple local features\.*Information Sciences*, 509:257–289, 2020\.
- Carlini et al\. \(2022\)Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer\.Membership inference attacks from first principles, 2022\.URL[https://arxiv\.org/abs/2112\.03570](https://arxiv.org/abs/2112.03570)\.
- Chang et al\. \(2024\)Tyler A\. Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, and Ian Tenney\.Scalable influence and fact tracing for large language model pretraining, 2024\.URL[https://arxiv\.org/abs/2410\.17413](https://arxiv.org/abs/2410.17413)\.
- Choquette\-Choo et al\. \(2021\)Christopher A\. Choquette\-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot\.Label\-only membership inference attacks, 2021\.URL[https://arxiv\.org/abs/2007\.14321](https://arxiv.org/abs/2007.14321)\.
- Cohen and Giryes \(2022\)Gilad Cohen and Raja Giryes\.Membership inference attack using self influence functions, 2022\.URL[https://arxiv\.org/abs/2205\.13680](https://arxiv.org/abs/2205.13680)\.
- Demszky et al\. \(2020\)Dorottya Demszky, Dana Movshovitz\-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi\.GoEmotions: A Dataset of Fine\-Grained Emotions\.In*58th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2020\.
- Dernoncourt and Lee \(2017\)Franck Dernoncourt and Ji Young Lee\.Pubmed 200k rct: a dataset for sequential sentence classification in medical abstracts\.In*International Joint Conference on Natural Language Processing*, 2017\.URL[https://arxiv\.org/abs/1710\.06071](https://arxiv.org/abs/1710.06071)\.
- Du et al\. \(2023\)Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu\.Shortcut learning of large language models in natural language understanding, 2023\.URL[https://arxiv\.org/abs/2208\.11857](https://arxiv.org/abs/2208.11857)\.
- Firca et al\. \(2025\)Liviu Nicolae Firca, Antonio Barbalau, Dan Oneata, and Elena Burceanu\.Not all splits are equal: Rethinking attribute generalization across unrelated categories\.*NeurIPSW*, 2025\.
- Ganguly \(2023\)Amrita Ganguly\.SMED: Social media emotion dataset\.Kaggle, 2023\.URL[https://www\.kaggle\.com/datasets/gangulyamrita/social\-media\-emotion\-dataset](https://www.kaggle.com/datasets/gangulyamrita/social-media-emotion-dataset)\.Five\-class emotion dataset curated from social media\.
- Geertzen et al\. \(2013\)Jeroen Geertzen, Theodora Alexopoulou, and Anna Korhonen\.The ef\-cambridge open language database \(efcamdat\)\.*Annual Review of Applied Linguistics*, 33:208–231, 2013\.
- Geirhos et al\. \(2020\)Robert Geirhos, Jörn\-Henrik Jacobsen, Claudio Michaelis, Richard S\. Zemel, Wieland Brendel, Matthias Bethge, and Felix A\. Wichmann\.Shortcut learning in deep neural networks\.*CoRR*, abs/2004\.07780, 2020\.URL[https://arxiv\.org/abs/2004\.07780](https://arxiv.org/abs/2004.07780)\.
- Gururangan et al\. \(2018\)Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R\. Bowman, and Noah A\. Smith\.Annotation artifacts in natural language inference data, 2018\.URL[https://arxiv\.org/abs/1803\.02324](https://arxiv.org/abs/1803.02324)\.
- Hudson and Jaf \(2018\)Thomas G Hudson and Sardar Jaf\.On the development of a large scale corpus for native language identification\.In*Proceedings of the 17th International Workshop on Treebanks and Linguistic Theories \(TLT 2018\), December 13–14, 2018, Oslo University, Norway*, volume 155 of*Linköping Electronic Conference Proceedings*, pages 115–129\. Linköping University Electronic Press, 2018\.
- Ilyas et al\. \(2022\)Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry\.Datamodels: Predicting predictions from training data, 2022\.URL[https://arxiv\.org/abs/2202\.00622](https://arxiv.org/abs/2202.00622)\.
- Izmailov et al\. \(2022\)Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson\.On feature learning in the presence of spurious correlations, 2022\.URL[https://arxiv\.org/abs/2210\.11369](https://arxiv.org/abs/2210.11369)\.
- Koh and Liang \(2020\)Pang Wei Koh and Percy Liang\.Understanding black\-box predictions via influence functions, 2020\.URL[https://arxiv\.org/abs/1703\.04730](https://arxiv.org/abs/1703.04730)\.
- Leino and Fredrikson \(2020\)Klas Leino and Matt Fredrikson\.Stolen memories: Leveraging model memorization for calibrated white\-box membership inference, 2020\.URL[https://arxiv\.org/abs/1906\.11798](https://arxiv.org/abs/1906.11798)\.
- Li and Zhang \(2021\)Zheng Li and Yang Zhang\.Membership leakage in label\-only exposures, 2021\.URL[https://arxiv\.org/abs/2007\.15528](https://arxiv.org/abs/2007.15528)\.
- Lin et al\. \(2024\)Manxi Lin, Nina Weng, Kamil Mikolaj, Zahra Bashir, Morten BS Svendsen, Martin G Tolsgaard, Anders N Christensen, and Aasa Feragen\.Shortcut learning in medical image segmentation\.In*International Conference on Medical Image Computing and Computer\-Assisted Intervention*, pages 623–633\. Springer, 2024\.
- Liu et al\. \(2022\)Yiyong Liu, Zhengyu Zhao, Michael Backes, and Yang Zhang\.Membership inference attacks by exploiting loss trajectory, 2022\.URL[https://arxiv\.org/abs/2208\.14933](https://arxiv.org/abs/2208.14933)\.
- Maini et al\. \(2024\)Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic\.Llm dataset inference: Did you train on my dataset?, 2024\.URL[https://arxiv\.org/abs/2406\.06443](https://arxiv.org/abs/2406.06443)\.
- McCoy et al\. \(2019\)R\. Thomas McCoy, Ellie Pavlick, and Tal Linzen\.Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference, 2019\.URL[https://arxiv\.org/abs/1902\.01007](https://arxiv.org/abs/1902.01007)\.
- Nasr et al\. \(2018\)Milad Nasr, Reza Shokri, and Amir Houmansadr\.Machine learning with membership privacy using adversarial regularization, 2018\.URL[https://arxiv\.org/abs/1807\.05852](https://arxiv.org/abs/1807.05852)\.
- Nasr et al\. \(2019\)Milad Nasr, Reza Shokri, and Amir Houmansadr\.Comprehensive privacy analysis of deep learning: Passive and active white\-box inference attacks against centralized and federated learning\.In*2019 IEEE Symposium on Security and Privacy \(SP\)*, page 739–753\. IEEE, May 2019\.doi:10\.1109/sp\.2019\.00065\.URL[http://dx\.doi\.org/10\.1109/SP\.2019\.00065](http://dx.doi.org/10.1109/SP.2019.00065)\.
- Niven and Kao \(2019\)Timothy Niven and Hung\-Yu Kao\.Probing neural network comprehension of natural language arguments, 2019\.URL[https://arxiv\.org/abs/1907\.07355](https://arxiv.org/abs/1907.07355)\.
- Öhman et al\. \(2020\)Emily Öhman, Marc Pàmies, Kaisla Kajava, and Jörg Tiedemann\.XED: A multilingual dataset for sentiment analysis and emotion detection\.In*The 28th International Conference on Computational Linguistics \(COLING 2020\)*, 2020\.
- Paduraru et al\. \(2026\)Cristian Daniel Paduraru, Antonio Barbalau, Radu Filipescu, Andrei Liviu Nicolicioiu, and Elena Burceanu\.Bridging explainability and embeddings: Bee aware of spuriousness\.In*ICLR*, 2026\.
- Papernot et al\. \(2018\)Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P\. Wellman\.Sok: Security and privacy in machine learning\.In*2018 IEEE European Symposium on Security and Privacy \(EuroS&P\)*, pages 399–414, 2018\.doi:10\.1109/EuroSP\.2018\.00035\.
- Park et al\. \(2023\)Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry\.Trak: Attributing model behavior at scale, 2023\.URL[https://arxiv\.org/abs/2303\.14186](https://arxiv.org/abs/2303.14186)\.
- Peng et al\. \(2024\)Yuefeng Peng, Jaechul Roh, Subhransu Maji, and Amir Houmansadr\.Oslo: One\-shot label\-only membership inference attacks, 2024\.URL[https://arxiv\.org/abs/2405\.16978](https://arxiv.org/abs/2405.16978)\.
- Pulle \(2025\)Shreyas Pulle\.20\-emotion text classification dataset, 2025\.URL[https://huggingface\.co/datasets/shreyaspulle98/emotion\-dataset\-20\-emotions](https://huggingface.co/datasets/shreyaspulle98/emotion-dataset-20-emotions)\.
- Rabinovich et al\. \(2018\)Ella Rabinovich, Yulia Bogdanova, Dimitra Laparidou, Svitlana Volkova, and Yulia Tsvetkov\.Native language cognate effects on second language lexical choice\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 2745–2755, Brussels, Belgium, October 2018\. Association for Computational Linguistics\.doi:10\.18653/v1/D18\-1296\.URL[https://aclanthology\.org/D18\-1296](https://aclanthology.org/D18-1296)\.
- Rei and Mladenić \(2023\)Luis Rei and Dunja Mladenić\.Detecting fine\-grained emotions in literature\.*Applied Sciences*, 13\(13\), 2023\.ISSN 2076\-3417\.doi:10\.3390/app13137502\.URL[https://www\.mdpi\.com/2076\-3417/13/13/7502](https://www.mdpi.com/2076-3417/13/13/7502)\.
- Rigaki and Garcia \(2023\)Maria Rigaki and Sebastian Garcia\.A survey of privacy attacks in machine learning\.*ACM Computing Surveys*, 56\(4\):1–34, November 2023\.ISSN 1557\-7341\.doi:10\.1145/3624010\.URL[http://dx\.doi\.org/10\.1145/3624010](http://dx.doi.org/10.1145/3624010)\.
- Sablayrolles et al\. \(2019\)Alexandre Sablayrolles, Matthijs Douze, Yann Ollivier, Cordelia Schmid, and Hervé Jégou\.White\-box vs black\-box: Bayes optimal strategies for membership inference, 2019\.URL[https://arxiv\.org/abs/1908\.11229](https://arxiv.org/abs/1908.11229)\.
- Salem et al\. \(2018\)Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes\.Ml\-leaks: Model and data independent membership inference attacks and defenses on machine learning models, 2018\.URL[https://arxiv\.org/abs/1806\.01246](https://arxiv.org/abs/1806.01246)\.
- Sculley \(2010\)David Sculley\.Web\-scale k\-means clustering\.In*Proceedings of the 19th International Conference on World Wide Web*, pages 1177–1178\. ACM, 2010\.
- Shokri et al\. \(2017\)Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov\.Membership inference attacks against machine learning models, 2017\.URL[https://arxiv\.org/abs/1610\.05820](https://arxiv.org/abs/1610.05820)\.
- Song and Mittal \(2020\)Liwei Song and Prateek Mittal\.Systematic evaluation of privacy risks of machine learning models, 2020\.URL[https://arxiv\.org/abs/2003\.10595](https://arxiv.org/abs/2003.10595)\.
- Spärck Jones \(1972\)Karen Spärck Jones\.A statistical interpretation of term specificity and its application in retrieval\.*Journal of Documentation*, 28\(1\):11–21, 1972\.
- Sun et al\. \(2025\)Weiwei Sun, Haokun Liu, Nikhil Kandpal, Colin Raffel, and Yiming Yang\.Enhancing training data attribution with representational optimization, 2025\.URL[https://arxiv\.org/abs/2505\.18513](https://arxiv.org/abs/2505.18513)\.
- Wang et al\. \(2025\)Jiachen T\. Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia\.Data shapley in one training run, 2025\.URL[https://arxiv\.org/abs/2406\.11011](https://arxiv.org/abs/2406.11011)\.
- Wang et al\. \(2021\)Tan Wang, Chang Zhou, Qianru Sun, and Hanwang Zhang\.Causal attention for unbiased visual recognition, 2021\.URL[https://arxiv\.org/abs/2108\.08782](https://arxiv.org/abs/2108.08782)\.
- Yang et al\. \(2022\)Yao\-Yuan Yang, Chi\-Ning Chou, and Kamalika Chaudhuri\.Understanding rare spurious correlations in neural networks, 2022\.URL[https://arxiv\.org/abs/2202\.05189](https://arxiv.org/abs/2202.05189)\.
- Ye et al\. \(2022\)Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri\.Enhanced membership inference attacks against machine learning models, 2022\.URL[https://arxiv\.org/abs/2111\.09679](https://arxiv.org/abs/2111.09679)\.
- Ye et al\. \(2024\)Wenqian Ye, Luyang Jiang, Eric Xie, Guangtao Zheng, Yunsheng Ma, Xu Cao, Dongliang Guo, Daiqing Qi, Zeyu He, Yijun Tian, et al\.The clever hans mirage: A comprehensive survey on spurious correlations in machine learning\.*arXiv preprint arXiv:2402\.12715*, 2024\.
- Yeom et al\. \(2018\)Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha\.Privacy risk in machine learning: Analyzing the connection to overfitting, 2018\.URL[https://arxiv\.org/abs/1709\.01604](https://arxiv.org/abs/1709.01604)\.
- Zarifzadeh et al\. \(2024\)Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri\.Low\-cost high\-power membership inference attacks, 2024\.URL[https://arxiv\.org/abs/2312\.03262](https://arxiv.org/abs/2312.03262)\.

## Appendix AAppendix

### A\.1Broader Impact

This work supports training\-data auditing by providing a white\-box signal for dataset attribution and dataset\-level membership inference\. Potential positive uses include dataset provenance analysis, benchmark contamination detection, licensing audits, and accountability for models trained on mixed data sources\. The method is also dual\-use\. The same signal could be used to infer confidential or commercially sensitive information about a model’s training data\. For this reason, the results should be interpreted as statistical evidence rather than definitive proof of dataset use, especially in legal, privacy\-sensitive, or compliance settings\.

### A\.2Dataset configurations and training regimes

Tab\.[1](https://arxiv.org/html/2605.30462#A1.T1)reports the dataset configurations used in our experiments\. For each dataset group, we construct three types of training sets: individual datasets, leave\-one\-dataset\-out mixtures, and the full combined corpus\. The individual configurations provide standalone dataset fingerprints, the leave\-one\-dataset\-out configurations support the controlled sensitivity analysis, and the full corpus configurations are used for dataset\-level membership evaluation\.

Overall, this yields 27 training configurations across the three experimental groups: 9 for emotion classification, 7 for natural language inference, and 11 for medical text classification\. The table also reports the number of training rows in each configuration, making explicit the scale of each model training regime used in the main experiments\.

Table 1:Dataset configurations and sample sizes across the three experimental groups\. For each group, we construct individual, leave\-one\-dataset\-out, and full\-corpus training sets, yielding 27 training configurations in total\.Dataset groupTraining configurationSample size \(rows\)EmotionEmoLit \(Individual\)200,727EmotionDataset20 \(Individual\)79,595GoEmotions \(Individual\)53,993XED & SMED \(Individual\)92,978Without EmoLit \(Leave\-One\-Out\)226,563Without EmotionDataset20 \(Leave\-One\-Out\)347,695Without GoEmotions \(Leave\-One\-Out\)373,300Without XED & SMED \(Leave\-One\-Out\)334,315Full Combined Corpus427,290NLIiTalki \(Individual\)45,204Reddit\-L2 \(Individual\)45,201EFCAMDAT \(Individual\)45,200Without iTalki \(Leave\-One\-Out\)90,401Without Reddit\-L2 \(Leave\-One\-Out\)90,404Without EFCAMDAT \(Leave\-One\-Out\)90,405Full Balanced Corpus135,605MedicalCluster 0 \(Individual\)21,100Cluster 1 \(Individual\)31,791Cluster 2 \(Individual\)33,251Cluster 3 \(Individual\)160,516Cluster 4 \(Individual\)25,776All Except Cluster 0 \(Leave\-One\-Out\)251,334All Except Cluster 1 \(Leave\-One\-Out\)240,643All Except Cluster 2 \(Leave\-One\-Out\)239,183All Except Cluster 3 \(Leave\-One\-Out\)111,918All Except Cluster 4 \(Leave\-One\-Out\)246,658Full Combined Corpus \(12% Subsample\)272,434
### A\.3Per\-class keyword overlap between single\-source datasets

To better understand why our membership inference attack performs differently across dataset groups, we measure how much the discriminative vocabularies of individual source datasets overlap\. A low overlap indicates that each source contributes a largely unique set of class\-discriminative features, leaving less shared signal for the fingerprint\-based detector to exploit when comparing models trained on different subsets\.

#### Procedure\.

For each dataset group \(NLI, Emotion, Medical\), we extract the top 5,000 keywords independently from each single\-source training set using YAKE \(withnn\-gram size 3 and deduplication threshold 0\.9\)\. For each single\-source configurationii, we then identify which of its keywords appear in the concatenated \(lowercased\) texts of each classcc, yielding a set of per\-class keyword occurrences\. We denote the number of keywords from configurationiithat appear in classccof its own dataset as part of this set\.

To quantify the overlap between two single\-source configurationsiiandjj, we count, for each classcc, how many keywords are found in the texts of classccin*both*configurations, and sum this count across all classes\. This gives the raw overlap value reported in the tables below as the off\-diagonal entries\. The diagonal entries represent the total number of per\-class keyword occurrences within each configuration’s own data \(i\.e\., self\-overlap\)\.

To obtain a normalized measure, we divide each pairwise overlap value by the average of the two corresponding diagonal \(self\-overlap\) values\. This yields a ratio between 0 and 1, where 1 would indicate identical per\-class keyword coverage and 0 would indicate completely disjoint vocabularies\. We then report the mean of all off\-diagonal normalized values as the average pairwise overlap for each dataset group\.

#### Results\.

Tab\.[2](https://arxiv.org/html/2605.30462#A1.T2)shows the normalized pairwise overlap matrix for the NLI group, Tab\.[3](https://arxiv.org/html/2605.30462#A1.T3)for the Emotion group, and Tab\.[4](https://arxiv.org/html/2605.30462#A1.T4)for the Medical group\. Tab\.[5](https://arxiv.org/html/2605.30462#A1.T5)summarizes the average pairwise overlap across groups alongside thei​dSCDid\_\{\\mathrm\{SCD\}\}ROC\-AUC\.

Table 2:Normalized per\-class keyword overlap between single\-source datasets \(NLI\)\.Table 3:Normalized per\-class keyword overlap between single\-source datasets \(Emotion\)\.Table 4:Normalized per\-class keyword overlap between single\-source datasets \(Medical\)\.Table 5:Average pairwise normalized keyword overlap \(single\-source pairs only\) andi​dSCDid\_\{\\mathrm\{SCD\}\}membership inference ROC\-AUC for each dataset group\.
#### Implications\.

The NLI group exhibits the lowest average pairwise overlap \(14\.96%\), meaning that its constituent sources \(reddit\_l2, efcamdat, italki\) share only about 15% of their class\-discriminative vocabulary\. By contrast, the Emotion and Medical groups share approximately 20% of theirs\. This aligns with the loweri​dSCDid\_\{\\mathrm\{SCD\}\}ROC\-AUC observed for NLI: when single\-source datasets have less overlapping discriminative features, the aligned keyword matrices used by the detector contain more zero\-filled \(uninformative\) positions, which dilutes the similarity signal that distinguishes member from non\-member configurations\. In other words, less lexical overlap between sources means less information is available to inform the dataset fingerprint, making the membership inference task harder\.

Importantly, this overlap can be measured*a priori*, before training any models or running the attack, using only the raw text data and an unsupervised keyword extraction method\. This suggests that keyword overlap between constituent sources could serve as a practical diagnostic for anticipating the difficulty of fingerprint\-based dataset membership inference in multi\-source settings\.