Probing Memorization of Tabular In-Context Learning

# Probing Memorization of Tabular In-Context Learning
Source: [https://arxiv.org/html/2606.31208](https://arxiv.org/html/2606.31208)
###### Abstract

Large tabular models \(LTMs\), i\.e\., tabular foundation models leveraging in\-context learning \(ICL\), achieve state\-of\-the\-art performance on tabular tasks\. While LLMs are known to unintentionally memorize training data, the memorization dynamics of LTMs remain largely unexplored\. We investigate the potential for parametric memorization in tabular ICL\. We introduceIclMem, a probing framework designed to separate context\-based predictions from parametric memorization\. Our zero\-information multiple\-choice context strips away valid contextual patterns to force the model to fall back on its parametric memory\. Our controlled fine\-tuning setup establishes membership ground truth and accounts for common pitfalls, e\.g\., distribution shift, feature contamination, base\-rate fallacy, and the pre\-trained base model acts as reference to calibrate for sample difficulty\. Our controlled evaluation on a leading real\-world\-trained LTM detects moderate memorization signals in 8 out of 10 tasks \(AUCup to0\.670\.67and TPR at1%1\\%FPR\>0\.1\>0\.1\)\. Notably, memorization signals are strongest for low\-cardinality and binary tasks\. However, they largely vanish under realistic training conditions\. Our findings show LTM memorization signals under specific circumstances \(single\-task fine\-tuning with fixed samples across many epochs and small query size\)\. To protect sensitive data, appropriate measures must be taken, which we discuss\.

Tabular FM, In\-context learning, Privacy, Membership inference

## 1Introduction

Tabular data dominate enterprise applications and domains such as healthcare and finance\(Chuiet al\.,[2018](https://arxiv.org/html/2606.31208#bib.bib168)\)\. Large tabular models \(LTMs\)\(Van Breugel and Van Der Schaar,[2024](https://arxiv.org/html/2606.31208#bib.bib186);[Hollmannet al\.,](https://arxiv.org/html/2606.31208#bib.bib143);[Spinaciet al\.,](https://arxiv.org/html/2606.31208#bib.bib145)\), i\.e\., tabular foundation models, achieve state\-of\-the\-art performance for tabular predictions by leveraging in\-context learning \(ICL\) to perform zero\-shot predictions on unseen tasks\(Brownet al\.,[2020](https://arxiv.org/html/2606.31208#bib.bib167)\)\. At inference, the model receives a context \(e\.g\., historical example rows\) alongside a query row with a missing target field to be predicted\. While LTMs are typically pre\-trained on synthetic data\([Hollmannet al\.,](https://arxiv.org/html/2606.31208#bib.bib143); Quet al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib144)\), recent models incorporate real\-world tabular corpora\([Spinaciet al\.,](https://arxiv.org/html/2606.31208#bib.bib145); Garget al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib169); Maet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib170)\), e\.g\., by fine\-tuning on diverse real\-world task mixtures to improve performance\(Grinsztajnet al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib189)\)\. While LLMs are known to unintentionally memorize training data\(Chenet al\.,[2026](https://arxiv.org/html/2606.31208#bib.bib179); Hayeset al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib153); Szepet al\.,[2026](https://arxiv.org/html/2606.31208#bib.bib177)\), the memorization dynamics of LTMs remain largely unexplored\. Rigorous memorization probing is crucial to inform potential mitigations for sensitive fine\-tuning data, such as differential privacy and adaptation of data pre\-processing and training regimes\.

Problem Setting\.

Traditional LLM ICLx1x\_\{1\}y1y\_\{1\}⋯\\cdotsxnx\_\{n\}yny\_\{n\}ContextCCModel ProviderQueryqqUserGoal: Infer ifq∈Cq\\in CLLMPredictiony^\\hat\{y\}LTM ICLx1x\_\{1\}y1y\_\{1\}⋯\\cdotsxnx\_\{n\}yny\_\{n\}ContextCCQueryqqUserGoal: Infer ifq∈Dq\\in DLTMMθM\_\{\\theta\}Fine\-tuned onDDPredictiony^\\hat\{y\}

Figure 1:Memorization probing for LLM vs LTM\.Left:In LLMs with ICL the model provider supplies a sensitive contextCC, which the user attempts to extract via user\-controlled queries\.Right:In LTMs, the user controls both the contextCCand the queryqq\. InIclMemthe context is used to probe whetherqqwas memorized from the sensitive fine\-tuning dataDD\.In the ICL paradigm, a model predicts a missing target valueyqy\_\{q\}given query’s input featuresxqx\_\{q\}, and a contextC=\{\(x1,y1\),…,\(xn,yn\)\}C=\\\{\(x\_\{1\},y\_\{1\}\),\\dots,\(x\_\{n\},y\_\{n\}\)\\\}of examples\. With LTMs, the user provides both the contextCCand the queryqq, as shown in Fig\.[1](https://arxiv.org/html/2606.31208#S1.F1)\. We assume an LTMMΘM\_\{\\Theta\}is fine\-tuned on sensitiveDDby a data owner who provides access to the fine\-tuned LTM to users\. This setting opens up the possibility to manipulateCCarbitrarily, to probe differences in model behavior on context\-query pairs derived from data included in or excluded fromDD\. In contrast, existing works on ICL privacy primarily assume the model provider defines a sensitiveCC, which an attacker attempts to extract via membership inference attacks \(MIA\) by manipulatingqq\(Wenet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib149); Duanet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib173)\)\. MIAs\(Shokriet al\.,[2017](https://arxiv.org/html/2606.31208#bib.bib40)\)aim to infer whether a specific record was part of the training data, and transferring MIAs to LTMs presents unique challenges\. Unlike LLMs, which can generate arbitrary text on a vast output space to extract memorized tokens\(Carliniet al\.,[2021](https://arxiv.org/html/2606.31208#bib.bib43)\), LTMs are encoder\-only architectures that strictly constrain predictions to the pre\-defined candidate labels provided inCC\([Spinaciet al\.,](https://arxiv.org/html/2606.31208#bib.bib145);[Hollmannet al\.,](https://arxiv.org/html/2606.31208#bib.bib143)\)\. More importantly, while masked\-language modeling can induce memorization\(Hartmannet al\.,[2023](https://arxiv.org/html/2606.31208#bib.bib190)\), LTMs operate as Prior\-Data Fitted Networks \(PFNs\)\(Mülleret al\.,[2021](https://arxiv.org/html/2606.31208#bib.bib181)\)\. They are trained to infer structural connections within the context rather than memorize explicit targets\(Mülleret al\.,[2021](https://arxiv.org/html/2606.31208#bib.bib181)\)\. This raises a fundamental, yet unanswered question:*What memorization profile do LTMs, inherently trained to perform ICL, have?*To investigate memorization capabilities of LTMs, we presentIclMemand apply MIA\. Specifically, we test membership of a candidate set of query rows by manipulatingCCas an active probing mechanism \(Fig\.[1](https://arxiv.org/html/2606.31208#S1.F1)\)\. We hypothesize that by carefully modifyingCCto eliminate discriminative information, it disrupts the model’s context\-derived prediction capabilities\. Forced to abandon in\-context reasoning, the model falls back on its memorized parametric knowledge\. For member rows inDD, we expect predictions to exhibit higher confidence and remain robust to context manipulations\(Choquette\-Chooet al\.,[2021](https://arxiv.org/html/2606.31208#bib.bib172); Yeomet al\.,[2018](https://arxiv.org/html/2606.31208#bib.bib41)\)\. We identify specific circumstances for memorization in LTMs \(Sec\.[4\.2](https://arxiv.org/html/2606.31208#S4.SS2)\) and discuss protection measures \(Sec\.[6](https://arxiv.org/html/2606.31208#S6)\)\.

## 2Related Work

While LTMs\([Hollmannet al\.,](https://arxiv.org/html/2606.31208#bib.bib143); Quet al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib144);[Spinaciet al\.,](https://arxiv.org/html/2606.31208#bib.bib145); Garget al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib169)\)rapidly advance, investigations into their privacy properties remain limited\(Nayyeriet al\.,[2026](https://arxiv.org/html/2606.31208#bib.bib180)\)\. Existing LTM privacy literature primarily targets test\-time evasion and adversarial robustness rather than memorization\(Simonettoet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib146); Djilaniet al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib147); Anwaret al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib148)\)\. Conversely, ICL privacy research largely focuses on LLMs, probing either the leakage of hidden context examples\(Wenet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib149); Duanet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib173)\)or table memorization within text corpora\(Germanet al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib150); Bordtet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib184)\)\. To assess memorization, state\-of\-the\-art MIAs typically rely on shadow models\(Carliniet al\.,[2022a](https://arxiv.org/html/2606.31208#bib.bib151); Zarifzadehet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib152)\)\. However, training shadow models for large models is computationally prohibitive\(Hayeset al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib153)\)and yields unreliable false\-positive bounds when the pre\-training distribution is unknown\(Zhanget al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib154)\)\. Towards privacy assessment of LTMs, we adapt techniques from robustness\-assessing MIA\(Choquette\-Chooet al\.,[2021](https://arxiv.org/html/2606.31208#bib.bib172)\)and attribute inference \(AIA\)\(Jayaraman and Evans,[2022](https://arxiv.org/html/2606.31208#bib.bib183); Annamalaiet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib156); Salemet al\.,[2023](https://arxiv.org/html/2606.31208#bib.bib37)\)\. While AIA aims to infer an unknown sensitive value given a partial record, our goal is inferring membership of a record\. To assess memorization in LTMs, we adapt AIA’s multiple\-choice testing and label randomization and combine zero\-information context manipulations with difficulty calibration via the pre\-trained base model\(Watsonet al\.,[2022](https://arxiv.org/html/2606.31208#bib.bib175)\)\.

## 3Contributions

We provide the first systematic assessment of memorization in LTMs\. Our contributions are:

- •We introduceIclMem, a probing framework designed to separate context\-based predictions from parametric memorization\. Our zero\-information multiple\-choice context strips away contextual patterns forcing the model to fall back on parametric memorization\.
- •We avoid common MIA false positives by accounting for distribution shifts \(via data pre\-processing\), feature contamination \(via data deduplication\), base\-rate confounding \(via label randomization\), and intrinsic low\-complexity samples \(via difficulty calibration\)\.
- •We perform an extensive evaluation on LTM ConTextTab across 1,128 fine\-tuning configurations on 10 diverse classification and regression tasks from CARTE\.
- •We detect moderate memorization in 8 of the 10 tasks \(withAUC\>0\.5\\text\{AUC\}\>0\.5andTPR@​r%\>0\.1\\text\{TPR\}\_\{@r\\%\}\>0\.1forr∈\{1,10\}r\\in\\\{1,10\\\}\) in our controlled setup with single\-task fine\-tuning and fixed context\-query pairs\. The signal is higher for low\-cardinality and binary tasks with small training query sizes \(Q=50Q=50\), but largely vanishes under practical setups \(e\.g\., largerQ=512Q=512, multi\-dataset training, random context\-query sampling\) \(App\.[D](https://arxiv.org/html/2606.31208#A4)\)\.

Overall, our findings show memorization capabilities in LTMs which help to inform appropriate protection measures\. Next, we introduce our methodology and setup \(Sec\.[4](https://arxiv.org/html/2606.31208#S4)\), detail our evaluation \(Sec\.[5](https://arxiv.org/html/2606.31208#S5)\), and conclude with limitations, mitigations, and future works \(Sec\.[6](https://arxiv.org/html/2606.31208#S6)\)\.

## 4Methodology and Setup

We introduceIclMem, a framework designed to probe memorization in LTMs and mitigate common MIA evaluation pitfalls \(Tab\.[4](https://arxiv.org/html/2606.31208#A1.T4), App\.[A\.1](https://arxiv.org/html/2606.31208#A1.SS1)\)\. First, we outline unique challenges of assessing LTMs\. Then, we discuss the two stages ofIclMem\(Fig\.[3](https://arxiv.org/html/2606.31208#A1.F3), App\.[A](https://arxiv.org/html/2606.31208#A1)\)\. Namely, controlled fine\-tuning with careful data pre\-processing and fixed query\-context pairs across epochs \(Sec\.[4\.1](https://arxiv.org/html/2606.31208#S4.SS1)\) and memorization probing with a suite of context manipulations calibrated against the pre\-trained base model as reference \(Sec\.[4\.2](https://arxiv.org/html/2606.31208#S4.SS2)\)\.

Challenges in Assessing LTMs\.LTMs rely on ICL to infer structural feature\-target correlations from the context, a mechanism that intuitively acts as a barrier to accessing the model’s parametric memory\. Furthermore, unlike LLMs with massive output vocabularies, LTMs are encoder\-only architectures strictly constrained to outputting probabilities over the specific candidate labels present in the context\. Our context manipulations address these challenges via zero\-information multiple\-choice protocol \(Sec\.[4\.2](https://arxiv.org/html/2606.31208#S4.SS2)\)\.

### 4\.1Establishing a Memorization Ground Truth

IclMempre\-processes data to eliminate artifacts, and performs controlled fine\-tuning to induce memorization\.

Data Pre\-Processing\.To avoid data artifacts being interpreted as memorization signal,IclMemcarefully pre\-processes data\. First, we deduplicate records to prevent*feature contamination*, i\.e\., identical records appearing in both member and non\-member sets, which inflate the memorization signal\. Second, we ensure distributional closeness between disjoint member and non\-member splits to preclude*distributional shifts*\. Specifically, for target label distributions via total variation distanceTVD<0\.05\\text\{TVD\}<0\.05\(Gibbs and Su,[2002](https://arxiv.org/html/2606.31208#bib.bib166)\)for classification and the Kolmogorov–Smirnov statisticKS<0\.05\\text\{KS\}<0\.05\(Massey Jr,[1951](https://arxiv.org/html/2606.31208#bib.bib165)\)for regression \(i\.e\., difference between empirical cumulative density functions\) \(Tab\.[6](https://arxiv.org/html/2606.31208#A1.T6), App\.[A\.2](https://arxiv.org/html/2606.31208#A1.SS2)\)\. To mitigate the*base\-rate fallacy*, i\.e\., model exploits label distribution to predict likely outcomes, we apply two measures\. First, we enforce a uniform label distribution in the training context, preventing the model from minimizing loss by collapsing to majority\-class predictions\. Second, following\(Annamalaiet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib156)\), we apply label randomization, i\.e\., scrambling target labels while preserving marginal distributions, forcing the model to learn statistically improbable mappings\.

Controlled Fine\-Tuning\.To induce memorization, we fine\-tune on fixed context\-query pairs for up to 500 epochs\. Specifically, each training step processes a static context setC′C^\{\\prime\}with a batch of query rows of sizeQQ\. We fine\-tune across varying learning ratesη∈\{10−1,10−2,10−3,10−4\}\\eta\\in\\\{10^\{\-1\},10^\{\-2\},10^\{\-3\},10^\{\-4\}\\\}and query sizesQ∈\{50,512\}Q\\in\\\{50,512\\\}\. We varyQQin training to test its influence on memorization\. SmallerQQshould amplify the contribution of individual query rows, while largerQQshould dilute them\. Our evaluation supports this dynamic, showing that largerQ=512Q=512hinders memorization \(Sec\.[5](https://arxiv.org/html/2606.31208#S5)\)\.

### 4\.2Probing Memorization withIclMem

The probing stage aims to detect memorization signals by combining zero\-information context manipulations with difficulty calibration via the base model\.

Multiple\-Choice Protocol\.To separate expected context\-based model predictions from unintended parametric memorization, we propose a zero\-information strategy mimicking a multiple\-choice question\. For each query rowq=\(xq,yq\)q=\(x\_\{q\},y\_\{q\}\)and candidate label setYY, we construct a probing contextCb​a​s​eC\_\{base\}containing exactly\|Y\|\|Y\|copies of the query’s featuresxqx\_\{q\}, each paired with a distinct candidate labelyi∈Yy\_\{i\}\\in Y, thereby providing the model with all possible candidate targets\. Our hypothesis is that if we strip away all valid contextual patterns, the model will be forced to abandon in\-context inference and default to its parametric memory, selecting the target value observed during fine\-tuning\.

Context Manipulations\.To assess robustness of predictions across multiple runs, we subjectCb​a​s​eC\_\{base\}to a suite of 48 context manipulationsΠ\\Pi, including positional biasing, distractor injection, and distributional shifts by, e\.g\., increasing the counts of true target labels, or non\-true labels \(detailed in App\.[A\.3](https://arxiv.org/html/2606.31208#A1.SS3)\)\. We compute the probe lossLk​\(q\)=ℒ​\(Mθ​\(xq∣Ck\),yq\)L\_\{k\}\(q\)=\\mathcal\{L\}\(M\_\{\\theta\}\(x\_\{q\}\\mid C\_\{k\}\),y\_\{q\}\)for each manipulated contextCk∈ΠC\_\{k\}\\in\\Pi\. The robustness score is defined as the average loss across all manipulationsS​\(q\)=1\|Π\|​∑πk∈ΠLk​\(q\)S\(q\)=\\frac\{1\}\{\|\\Pi\|\}\\sum\_\{\\pi\_\{k\}\\in\\Pi\}L\_\{k\}\(q\), which is the basis for our memorization metric\.

Difficulty Calibration\.An inherently easy query \(e\.g\., with obvious target from its features\) has a low loss, regardless of its membership inDD\. To account for this, we follow the*difficulty calibration*approach of Watson et al\.\(Watsonet al\.,[2022](https://arxiv.org/html/2606.31208#bib.bib175)\)and compare robustness score from the fine\-tuned model,S​\(q\)S\(q\), to that of the pre\-trained base model,Sr​e​f​\(q\)S\_\{ref\}\(q\), and compute its delta asΔ​S​\(q\)=S​\(q\)−Sr​e​f​\(q\)\\Delta S\(q\)=S\(q\)\-S\_\{ref\}\(q\)\. Notably, we computeS​\(q\)S\(q\)not only using the loss but also using distributional metrics, i\.e\., prediction entropy and target confidence, which remain robust discriminators even with fine\-tuning\-induced artifacts \(e\.g\., loss inversion, Sec\.[5](https://arxiv.org/html/2606.31208#S5)\)\.

Memorization Metric\.We compute AUC over theΔ​S\\Delta Sscores \(denotedAUC​\(Δ​S\)\\text\{AUC\}\(\\Delta S\)or simplyAUC\), and considerAUC\>0\.5\\text\{AUC\}\>0\.5as the threshold for*detectable*memorization\. Unlike high AUC baselines for LLMs\(Duanet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib173)\), LTMs operate over a constrained output space\. Hence, we adopt this threshold to capture any calibrated signal above chance\(Liuet al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib187)\)\. Furthermore, since global AUC averages across all records and can obscure localized memorization\(Carliniet al\.,[2022a](https://arxiv.org/html/2606.31208#bib.bib151)\), we report the true positive rate at low false positive rates \(TPR@​r%\\text\{TPR\}\_\{@r\\%\}forr∈\{1,10\}r\\in\\\{1,10\\\}\) providing a granular view of more confident signals\.

## 5Empirical Evaluation

Next, we summarize our results ofIclMem, and defer full details to App\.[C](https://arxiv.org/html/2606.31208#A3)due to space constraints\.

Setup\.We evaluateIclMemon ConTextTab\([Spinaciet al\.,](https://arxiv.org/html/2606.31208#bib.bib145)\)on 10 tasks from CARTE\(Kimet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib160)\)\. We select these tasks as they were not used in pre\-training and represent a wide variety of domains and data distributions \(details in Tab\.[5](https://arxiv.org/html/2606.31208#A1.T5)\)\. We test for memorization across epochs to capture the evolution of the memorization over time resulting in 1128 probing runs \(configurations in Tab\.[2](https://arxiv.org/html/2606.31208#A1.T2)\)\.

Isolating Memorization Outcomes\.Our evaluation yields five distinct outcomes \(summarized in Tab\.[3](https://arxiv.org/html/2606.31208#A1.T3)\)\.

Table 1:IclMemdetectable memorization configurations \(Q=50 except†withQ=512Q=512\)\. Full results in Tabs\.[10](https://arxiv.org/html/2606.31208#A3.T10)–[11](https://arxiv.org/html/2606.31208#A3.T11)in App\.[C](https://arxiv.org/html/2606.31208#A3)\.DatasetLREp\.AUC​\(Δ​S\)\\text\{AUC\}\(\\Delta S\)TPR@​10%\\text\{TPR\}\_\{@10\\%\}TPR@​1%\\text\{TPR\}\_\{@1\\%\}buy\_buy\_baby10−410^\{\-4\}100\.610\.610\.140\.140\.120\.12chocolate\_bar10−310^\{\-3\}100\.610\.610\.180\.180\.020\.02babies\_r\_us10−410^\{\-4\}500\.66\\mathbf\{0\.66\}0\.020\.020\.000\.00beer\_ratings†10−410^\{\-4\}500\.590\.590\.49\\mathbf\{0\.49\}0\.17\\mathbf\{0\.17\}bikedekho†10−410^\{\-4\}100\.580\.580\.150\.150\.020\.02Impractical fine\-tuning configurations \(e\.g\., large learning rate\) can induce data artifacts mimicking memorization\. The most prominent is*representational collapse*\(affecting≈55%\{\\approx\}55\\%of runs atQ=50Q=50\), where aggressive learning rates \(η≥10−2\\eta\\geq 10^\{\-2\}\) destroy predictive capabilities, yielding near\-constant outputs that artificially inflate loss\. Difficulty calibration identifies these false positives: ifMrefM\_\{\\text\{ref\}\}achieves identical zero\-shot separation, the signal reflects pre\-existing dataset bias, not memorization\. A secondary artifact is*loss inversion*\(≈19%\{\\approx\}19\\%of runs\), where pre\-trained structural biases drive member loss higher than non\-member loss\. A detailed breakdown of all outcomes, including context overfitting, is deferred to App\.[B\.2](https://arxiv.org/html/2606.31208#A2.SS2), with results for all datasets in Tabs\.[10](https://arxiv.org/html/2606.31208#A3.T10)–[11](https://arxiv.org/html/2606.31208#A3.T11)\. After accounting for these fine\-tuning artifacts, we isolate memorization signal in12\.0%12\.0\\%of runs atQ=50Q=50\(Tab\.[5](https://arxiv.org/html/2606.31208#A2.F5)\), and 8 of the 10 evaluated tasks in total under typical fine\-tuning configurations \(η≤10−3\\eta\\leq 10^\{\-3\}\), exemplified in Tab\.[1](https://arxiv.org/html/2606.31208#S5.T1)\. More granularly, the TPR at low FPR metric reveals localized memorization\. Onbeer\_ratingsatQ=512Q=512,IclMemachieves onlyAUC=0\.59\\text\{AUC\}=0\.59but identifies 49% of member records at 10% FPR; this dataset retains detectable memorization atQ=512Q=512due to its high target cardinality and small size, as analyzed below\. Onbuy\_buy\_baby, we reachTPR@​1%=0\.12\\text\{TPR\}\_\{@1\\%\}=0\.12, indicating that in some instances fine\-tuning records can be confidently distinguished\.

Drivers of Memorization\.We find that memorization depends on the interaction between query sizeQQ, target cardinalitykk, and dataset sizenn\. Firstly, expandingQQmitigates memorization in our evaluation: atQ=512Q=512, the overall detectable rate drops to1\.8%1\.8\\%\(a6\.7×6\.7\\timesreduction versusQ=50Q=50\), as larger queries stabilize batch variance and enforce stronger zero\-shot inductive biases\. Secondly, at low query sizes \(Q=50Q=50\), memorization is driven by*label density*\(n/kn/k, the average records per class\)\. Binary tasks \(k=2k=2, e\.g\.,chocolate\_bar\_ratings\) concentrate the gradient signal for specific classes per step, yielding the highest vulnerability \(up to55%55\\%detectable runs\)\. Askkgrows, this concentration dissolves\. Finally, high\-cardinality tasks require two conditions to trigger memorization: high*query coverage*\(Q/nQ/n, the dataset fraction seen per step\) and a target that is*not structurally deducible*\. If a target cannot be logically inferred from its features alone \(e\.g\., exact prices or subjective float ratings inbeer\_ratings\), the model is forced to fall back on parametric memory, providedQQcovers enough of the dataset\. Conversely, if a target can be deduced via ICL \(e\.g\., text\-statistic formulas inclear\_corpus\), the model relies on in\-context reasoning and yields no memorization signal, regardless ofQQ\.

112210105050100100250250500500112255101020205050Fine\-Tuning Epochs% of ConfigurationsCollapseInversionDet\. Mem\.Ctx\. Ovfit\.No SignalQ=50Q=50Q=512Q=512

Figure 2:Temporal evolution ofIclMemoutcomesTemporal Evolution of Memorization\.Across fine\-tuning epochs \(Fig\.[2](https://arxiv.org/html/2606.31208#S5.F2)\), theQ=50Q=50signal emerges at epoch 1 \(4\.2%4\.2\\%of configurations\) and grows to20%20\\%by epoch 250\. AtQ=512Q=512, memorization is largely suppressed: it peaks at5\.5%5\.5\\%around epoch 50 and vanishes after, showing that larger queries reduce memorization and accelerate its decay\.

## 6Discussion and Conclusion

Next, we discuss limitations, mitigations, and future work\.

Limitations\.First, whileIclMemdetects memorization, the signal is moderate: only 20% of tasks showTPR@​1%\>0\.1\\text\{TPR\}\_\{@1\\%\}\>0\.1\. Second, our probe uses an artificial setup \(single task, fixed context\-query pairs\) to induce memorization\. In contrast, realistic LTM regimes pre\-train on large task mixtures for only 2–5 epochs\([Spinaciet al\.,](https://arxiv.org/html/2606.31208#bib.bib145)\), which dilutes memorization\. We empirically verify that under joint multi\-task training with random context\-query sampling atQ=512Q=512, detectable tasks drop from 5 to 1, with the signal appearing only at epoch 10, i\.e\., beyond realistic budgets \(App\.[D](https://arxiv.org/html/2606.31208#A4)\)\. Finally, while our evaluation is broad, our scope is currently limited \(1 model, 10 tasks\) and to be expanded\.

Mitigations\.To protect sensitive data, appropriate measures must be taken\. Differential privacy \(e\.g\., DP\-SGD\(Abadiet al\.,[2016](https://arxiv.org/html/2606.31208#bib.bib8)\)\) provides rigorous privacy guarantees that requires careful privacy\-utility trade\-offs but also shows promising utility when fine\-tuning a public base model\(Yuet al\.,[2021](https://arxiv.org/html/2606.31208#bib.bib49); Liet al\.,[2021](https://arxiv.org/html/2606.31208#bib.bib51)\)\. Additionally, our findings show LTM memorization in certain training conditions that can be avoided \(e\.g\., small query size with fixed query\-context pairs over many epochs on single task, App\.[D](https://arxiv.org/html/2606.31208#A4)\) or inform data filtering of memorization\-prone samples \(albeit with potential for privacy onion effects\(Carliniet al\.,[2022b](https://arxiv.org/html/2606.31208#bib.bib188)\)\)\.

Conclusion and Future Work\.We initiated the systematic study of memorization in LTMs\. In our controlled setupIclMemdetects moderate memorization signal in 8 out of 10 evaluated tasks \(AUCup to0\.670\.67and TPR at1%1\\%FPR\>0\.1\>0\.1\)\. Our results demonstrate that, under specific conditions, tabular foundation models can memorize data, underscoring the need for appropriate protective measures when sensitive data is involved\. In future work, we aim to expand the evaluation to other LTMs\(Quet al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib144);[Hollmannet al\.,](https://arxiv.org/html/2606.31208#bib.bib143)\)and datasets\([Kleinet al\.,](https://arxiv.org/html/2606.31208#bib.bib159)\)\. We also aim to design*data canaries*\(i\.e\., crafted samples with unique fingerprints or improbable feature\-label mappings\(Carliniet al\.,[2019](https://arxiv.org/html/2606.31208#bib.bib185)\)\) to amplify memorization signal and bound the worst\-case detection limits ofIclMem\.

## References

- M\. Abadi, A\. Chu, I\. Goodfellow, H\. B\. McMahan, I\. Mironov, K\. Talwar, and L\. Zhang \(2016\)Deep learning with differential privacy\.CCS\.Cited by:[§6](https://arxiv.org/html/2606.31208#S6.p3.1)\.
- M\. S\. M\. S\. Annamalai, A\. Gadotti, and L\. Rocher \(2024\)A linear reconstruction approach for attribute inference attacks against synthetic data\.InUSENIX Security 24,Cited by:[§2](https://arxiv.org/html/2606.31208#S2.p1.1),[§4\.1](https://arxiv.org/html/2606.31208#S4.SS1.p2.2)\.
- U\. Anwar, J\. Von Oswald, L\. Kirsch, D\. Krueger, and S\. Frei \(2024\)Adversarial robustness of in\-context learning in transformers for linear regression\.Cited by:[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- S\. Bordt, H\. Nori, and R\. Caruana \(2024\)Elephants never forget: testing language models for memorization of tabular data\.arXiv preprint arXiv:2403\.06644\.Cited by:[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p1.1)\.
- N\. Carlini, S\. Chien, M\. Nasr, S\. Song, A\. Terzis, and F\. Tramer \(2022a\)Membership inference attacks from first principles\.In2022 IEEE symposium on security and privacy \(SP\),Cited by:[§2](https://arxiv.org/html/2606.31208#S2.p1.1),[§4\.2](https://arxiv.org/html/2606.31208#S4.SS2.p5.6)\.
- N\. Carlini, M\. Jagielski, C\. Zhang, N\. Papernot, A\. Terzis, and F\. Tramer \(2022b\)The privacy onion effect: memorization is relative\.Advances in Neural Information Processing Systems35\.Cited by:[§6](https://arxiv.org/html/2606.31208#S6.p3.1)\.
- N\. Carlini, C\. Liu, Ú\. Erlingsson, J\. Kos, and D\. Song \(2019\)The secret sharer: evaluating and testing unintended memorization in neural networks\.InUSENIX,Cited by:[§6](https://arxiv.org/html/2606.31208#S6.p4.4)\.
- N\. Carlini, F\. Tramer, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, U\. Erlingsson,et al\.\(2021\)Extracting training data from large language models\.USENIXSec\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p3.15)\.
- Y\. Chen, K\. Zhang, Y\. Du, E\. Stoppa, C\. Fleming, A\. Kundu, B\. Ribeiro, and N\. Li \(2026\)Membership inference attacks against fine\-tuned diffusion language models\.arXiv preprint arXiv:2601\.20125\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p1.1)\.
- C\. A\. Choquette\-Choo, F\. Tramer, N\. Carlini, and N\. Papernot \(2021\)Label\-only membership inference attacks\.InInternational conference on machine learning,Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p3.15),[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- M\. Chui, J\. Manyika, M\. Miremadi, N\. Henke, R\. Chung, P\. Nel, and S\. Malhotra \(2018\)Notes from the ai frontier: insights from hundreds of use cases\.McKinsey Global Institute\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p1.1)\.
- M\. Djilani, T\. Simonetto, K\. Tit, F\. Tambon, P\. Récamier, S\. Ghamizi, M\. Cordy, and M\. Papadakis \(2025\)On the robustness of tabular foundation models: test\-time attacks and in\-context defenses\.arXiv preprint arXiv:2506\.02978\.Cited by:[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- H\. Duan, A\. Dziedzic, M\. Yaghini, N\. Papernot, and F\. Boenisch \(2024\)On the privacy risk of in\-context learning\.arXiv preprint arXiv:2411\.10512\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p3.15),[§2](https://arxiv.org/html/2606.31208#S2.p1.1),[§4\.2](https://arxiv.org/html/2606.31208#S4.SS2.p5.6)\.
- J\. Evertz, N\. Risse, N\. Neuer, A\. Müller, P\. Normann, G\. Sapia, S\. Gupta, D\. Pape, S\. Shaw, D\. Srivastav,et al\.\(2025\)Chasing shadows: pitfalls in llm security research\.arXiv preprint arXiv:2512\.09549\.Cited by:[§A\.1](https://arxiv.org/html/2606.31208#A1.SS1.p1.1),[Table 4](https://arxiv.org/html/2606.31208#A1.T4),[Table 4](https://arxiv.org/html/2606.31208#A1.T4.14.2)\.
- A\. Garg, M\. Ali, N\. Hollmann, L\. Purucker, S\. Müller, and F\. Hutter \(2025\)Real\-tabpfn: improving tabular foundation models via continued pre\-training with real\-world data\.arXiv preprint arXiv:2507\.03971\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p1.1),[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- E\. German, S\. Antebi, D\. Samira, A\. Shabtai, and Y\. Elovici \(2025\)Tab\-mia: a benchmark dataset for membership inference attacks on tabular data in llms\.arXiv preprint arXiv:2507\.17259\.Cited by:[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- A\. L\. Gibbs and F\. E\. Su \(2002\)On choosing and bounding probability metrics\.International statistical review\.Cited by:[§A\.2](https://arxiv.org/html/2606.31208#A1.SS2.p2.10),[§4\.1](https://arxiv.org/html/2606.31208#S4.SS1.p2.2)\.
- L\. Grinsztajn, K\. Flöge, O\. Key, F\. Birkel, P\. Jund, B\. Roof, B\. Jäger, D\. Safaric, S\. Alessi, A\. Hayler,et al\.\(2025\)Tabpfn\-2\.5: advancing the state of the art in tabular foundation models\.arXiv preprint arXiv:2511\.08667\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p1.1)\.
- V\. Hartmann, A\. Suri, V\. Bindschaedler, D\. Evans, S\. Tople, and R\. West \(2023\)Sok: memorization in general\-purpose large language models\.arXiv preprint arXiv:2310\.18362\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p3.15)\.
- J\. Hayes, I\. Shumailov, C\. A\. Choquette\-Choo, M\. Jagielski, G\. Kaissis, K\. Lee, M\. Nasr, S\. Ghalebikesabi, N\. Mireshghallah, M\. Sundaram Mutu Selva Annamalai,et al\.\(2025\)Strong membership inference attacks on massive datasets and \(moderately\) large language models\.arXiv e\-prints\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p1.1),[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- \[22\]N\. Hollmann, S\. Müller, K\. Eggensperger, and F\. HutterTabPFN: a transformer that solves small tabular classification problems in a second\.InNeurIPS 2022 First Table Representation Workshop,Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p1.1),[§1](https://arxiv.org/html/2606.31208#S1.p3.15),[§2](https://arxiv.org/html/2606.31208#S2.p1.1),[§6](https://arxiv.org/html/2606.31208#S6.p4.4)\.
- B\. Jayaraman and D\. Evans \(2022\)Are attribute inference attacks just imputation?\.InACM SIGSAC,Cited by:[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- M\. J\. Kim, L\. Grinsztajn, and G\. Varoquaux \(2024\)CARTE: pretraining and transfer for tabular learning\.InInternational Conference on Machine Learning,Cited by:[§A\.2](https://arxiv.org/html/2606.31208#A1.SS2.p1.1),[Table 5](https://arxiv.org/html/2606.31208#A1.T5),[Table 5](https://arxiv.org/html/2606.31208#A1.T5.3.2),[§5](https://arxiv.org/html/2606.31208#S5.p2.1)\.
- \[25\]T\. Klein, C\. Biehl, M\. Costa, A\. Sres, J\. Kolk, and J\. HoffartSALT: sales autocompletion linked business tables dataset\.InNeurIPS 2024 Third Table Representation Learning Workshop,Cited by:[§6](https://arxiv.org/html/2606.31208#S6.p4.4)\.
- X\. Li, F\. Tramer, P\. Liang, and T\. Hashimoto \(2021\)Large language models can be strong differentially private learners\.arXiv preprint arXiv:2110\.05679\.Cited by:[§6](https://arxiv.org/html/2606.31208#S6.p3.1)\.
- D\. Liu, E\. Cohen, B\. Ghazi, P\. Kairouz, P\. Kamath, A\. Knop, R\. Kumar, P\. Manurangsi, A\. Sealfon, D\. Yu,et al\.\(2025\)URANIA: differentially private insights into ai use\.arXiv preprint arXiv:2506\.04681\.Cited by:[§4\.2](https://arxiv.org/html/2606.31208#S4.SS2.p5.6)\.
- J\. Ma, V\. Thomas, R\. Hosseinzadeh, A\. Labach, H\. Kamkari, J\. C\. Cresswell, K\. Golestan, G\. Yu, A\. L\. Caterini, and M\. Volkovs \(2024\)Tabdpt: scaling tabular foundation models on real data\.arXiv preprint arXiv:2410\.18164\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p1.1)\.
- B\. Marek, L\. Rossi, V\. Hanke, X\. Wang, M\. Backes, F\. Boenisch, and A\. Dziedzic \(2025\)Benchmarking empirical privacy protection for adaptations of large language models\.Cited by:[§A\.2](https://arxiv.org/html/2606.31208#A1.SS2.p2.10)\.
- F\. J\. Massey Jr \(1951\)The kolmogorov\-smirnov test for goodness of fit\.Journal of the American statistical Association\.Cited by:[§A\.2](https://arxiv.org/html/2606.31208#A1.SS2.p2.10),[§4\.1](https://arxiv.org/html/2606.31208#S4.SS1.p2.2)\.
- S\. Müller, N\. Hollmann, S\. P\. Arango, J\. Grabocka, and F\. Hutter \(2021\)Transformers can do bayesian inference\.arXiv preprint arXiv:2112\.10510\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p3.15)\.
- M\. Nayyeri, R\. B\. Thapa, D\. Hernández, A\. Arun, and S\. Staab \(2026\)A survey of structured data foundation models: a unified view on foundation models for tables, relational databases and knowledge graphs\.OpenReview\.External Links:[Link](https://openreview.net/forum?id=UbyITP0qq3)Cited by:[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. Le Morvan \(2025\)TabICL: a tabular foundation model for in\-context learning on large data\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p1.1),[§2](https://arxiv.org/html/2606.31208#S2.p1.1),[§6](https://arxiv.org/html/2606.31208#S6.p4.4)\.
- A\. Salem, G\. Cherubin, D\. Evans, B\. Köpf, A\. Paverd, A\. Suri, S\. Tople, and S\. Zanella\-Béguelin \(2023\)SoK: let the privacy games begin\! a unified treatment of data inference privacy in machine learning\.SP\.Cited by:[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- R\. Shokri, M\. Stronati, C\. Song, and V\. Shmatikov \(2017\)Membership inference attacks against machine learning models\.SP\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p3.15)\.
- T\. Simonetto, S\. Ghamizi, and M\. Cordy \(2024\)Constrained adaptive attack: effective adversarial attack against deep neural networks for tabular data\.Advances in Neural Information Processing Systems\.Cited by:[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- \[37\]M\. Spinaci, M\. Polewczyk, M\. Schambach, and S\. ThelinConTextTab: a semantics\-aware tabular in\-context learner\.In1st ICML Workshop on Foundation Models for Structured Data,Cited by:[Appendix D](https://arxiv.org/html/2606.31208#A4.p1.3),[§1](https://arxiv.org/html/2606.31208#S1.p1.1),[§1](https://arxiv.org/html/2606.31208#S1.p3.15),[§2](https://arxiv.org/html/2606.31208#S2.p1.1),[§5](https://arxiv.org/html/2606.31208#S5.p2.1),[§6](https://arxiv.org/html/2606.31208#S6.p2.2)\.
- M\. Szep, J\. M\. Ruiz, G\. Kaissis, P\. Seidl, R\. von Eisenhart\-Rothe, F\. Hinterwimmer, and D\. Rueckert \(2026\)Unintended memorization of sensitive information in fine\-tuned language models\.arXiv preprint arXiv:2601\.17480\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p1.1)\.
- B\. Van Breugel and M\. Van Der Schaar \(2024\)Why tabular foundation models should be a research priority\.arXiv preprint arXiv:2405\.01147\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p1.1)\.
- L\. Watson, C\. Guo, G\. Cormode, and A\. Sablayrolles \(2022\)On the importance of difficulty calibration in membership inference attacks\.InInternational Conference on Learning Representations,Cited by:[§A\.4](https://arxiv.org/html/2606.31208#A1.SS4.p1.1),[Table 4](https://arxiv.org/html/2606.31208#A1.T4.3.3.3.3.3),[§2](https://arxiv.org/html/2606.31208#S2.p1.1),[§4\.2](https://arxiv.org/html/2606.31208#S4.SS2.p4.5)\.
- R\. Wen, Z\. Li, M\. Backes, and Y\. Zhang \(2024\)Membership inference attacks against in\-context learning\.InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security,Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p3.15),[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- S\. Yeom, I\. Giacomelli, M\. Fredrikson, and S\. Jha \(2018\)Privacy risk in machine learning: analyzing the connection to overfitting\.CSF\.Cited by:[§1](https://arxiv.org/html/2606.31208#S1.p3.15)\.
- D\. Yu, S\. Naik, A\. Backurs, S\. Gopi, H\. A\. Inan, G\. Kamath, J\. Kulkarni, Y\. T\. Lee, A\. Manoel, L\. Wutschitz,et al\.\(2021\)Differentially private fine\-tuning of language models\.ICLR\.Cited by:[§6](https://arxiv.org/html/2606.31208#S6.p3.1)\.
- S\. Zarifzadeh, P\. Liu, and R\. Shokri \(2024\)Low\-cost high\-power membership inference attacks\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.
- J\. Zhang, D\. Das, G\. Kamath, and F\. Tramèr \(2025\)Position: membership inference attacks cannot prove that a model was trained on your data\.In2025 IEEE Conference on Secure and Trustworthy Machine Learning \(SaTML\),Cited by:[Table 4](https://arxiv.org/html/2606.31208#A1.T4),[Table 4](https://arxiv.org/html/2606.31208#A1.T4.14.2),[Table 4](https://arxiv.org/html/2606.31208#A1.T4.3.8.2.1.1),[§2](https://arxiv.org/html/2606.31208#S2.p1.1)\.

## Appendix ADetailed Probing Methodology and Pipeline

RawdataDeduplication &split withstatistical testsLabelrandomizationMembers\(IN\)Fine\-tuneLTMRecord pairs ofcontext\-queryNon\-members\(OUT\)Multiple\-choiceprotocolContextmanipulation \(Π\\Pi\)ModelinferenceDifficultyCalibrationMIAAUC\-ROCTraining context\(vanilla\)\(I\) Data Pre\-Processing\(II\) Controlled Fine\-Tuning\(III\) Multiple\-choice protocol\(IV\) Inference & Difficulty CalibrationStage 1: Establishing a Memorization Ground Truth \(Sec\.[4\.1](https://arxiv.org/html/2606.31208#S4.SS1)\)Stage 2: Probing Memorization \(Sec\.[4\.2](https://arxiv.org/html/2606.31208#S4.SS2)\)

Figure 3:End\-to\-endIclMempipeline\.Stage 1: \(I\)*Data pre\-processing*— deduplication, statistical split validation, member/non\-member assignment, and optional label randomization as a null\-hypothesis control \(applicable to members, non\-members, or both\); \(II\)*Controlled fine\-tuning*— LTM fine\-tuning on static context\-query pairs and logging of all context\-query pairs for verified ground truth\.Stage 2: \(III\)*Multiple\-choice protocol*— zero\-information context construction and adversarial context manipulation suiteΠ\\Pi; \(IV\)*Inference & difficulty calibration*— model inference under both the training \(vanilla\) context andΠ\\Pi, difficulty calibration with instance\-level delta scoreΔ​S​\(q\)=S​\(q\)−Sr​e​f​\(q\)\\Delta S\(q\)=S\(q\)\-S\_\{ref\}\(q\)computation against the pre\-trained reference modelMr​e​fM\_\{ref\}\.Figure[3](https://arxiv.org/html/2606.31208#A1.F3)illustrates theIclMempipeline, which consists of two main stages\. \(I\) Establishing a memorization ground truth via data pre\-processing and controlled fine\-tuning\. Here, we create a verified member/non\-member split, optionally randomize labels as a null\-hypothesis control, and log all context\-query pairs for ground\-truth verification\. \(II\) Probing memorization using a multiple\-choice protocol with adversarial context manipulation\. Here, we build a zero\-information context, apply a suite of 48 context manipulations, perform model inference under both the vanilla and manipulated contexts, aggregate scores, and compute delta scores against a pre\-trained reference model for difficulty calibration\.

Table 2:Experimental hyperparameters for LTM fine\-tuning and theIclMemprotocol\.StageParameterValue\(s\)Fine\-TuningBase ModelConTextTabOptimizerAdamWLearning Rates \(η\\eta\)\{10−1,10−2,10−3,10−4\}\\\{10^\{\-1\},10^\{\-2\},10^\{\-3\},10^\{\-4\}\\\}Maximum Epochs500Max context size512 rowsContext label distributionUniformQuery size \(QQ\)\{50,512\}\\\{50,512\\\}rowsQuery label distributionSource distributionMember label randomization\{False, True\}ProbingReference model \(Mr​e​fM\_\{ref\}\)Pre\-trainedConTextTabEpochs used for probing\{0, 1, 2, 5, 10, 50, 100, 250, 500\}Context size for probingDepends on the number of unique labels and adversarial manipulationsQuery size for probing1 query row replicated 50 timesAdversarial manipulations \(\|Π\|\|\\Pi\|\)48 distinct configurations \(see Tab\.[7](https://arxiv.org/html/2606.31208#A1.T7)\)Label randomization\{False, True, Only Members\}Evaluation metricsCross\-entropy loss, entropy, confidence,R2R^\{2\}Primary summary metricAUC​\(Δ​S\)\\text\{AUC\}\(\\Delta S\), whereΔ​S​\(q\)=S​\(q\)−Sr​e​f​\(q\)\\Delta S\(q\)=S\(q\)\-S\_\{ref\}\(q\)Tab\.[2](https://arxiv.org/html/2606.31208#A1.T2)lists all fine\-tuning and probing hyperparameters used in our evaluation\. Tab\.[3](https://arxiv.org/html/2606.31208#A1.T3)defines the classification criteria for the five evaluation outcomes\.

Table 3:Classification criteria for the five evaluation outcome categories, applied in priority order \(top to bottom\)\.AUC​\(Δ​S\)\\text\{AUC\}\(\\Delta S\)is computed on per\-instance delta scoresΔ​S​\(q\)=S​\(q\)−Sr​e​f​\(q\)\\Delta S\(q\)=S\(q\)\-S\_\{ref\}\(q\);ℒv​\(M,q\)\\mathcal\{L\}\_\{v\}\(M,q\)denotes the*vanilla loss*\(loss on the unmanipulated context, without anyπk\\pi\_\{k\}\);μgap\\mu\_\{\\text\{gap\}\}andσgap\\sigma\_\{\\text\{gap\}\}are the mean and standard deviation of the per\-manipulation probe loss gapgk=ℓ¯atkM​\(Mθ,πk\)−ℓ¯atkN​M​\(Mθ,πk\)g\_\{k\}=\\bar\{\\ell\}^\{M\}\_\{\\text\{atk\}\}\(M\_\{\\theta\},\\pi\_\{k\}\)\-\\bar\{\\ell\}^\{NM\}\_\{\\text\{atk\}\}\(M\_\{\\theta\},\\pi\_\{k\}\)across allπk∈Π\\pi\_\{k\}\\in\\Pi\.OutcomeKey ConditionIntuitionRepresentational Collapseℒv​\(Mθ\)\>5×ℒv​\(Mr​e​f\)\\mathcal\{L\}\_\{v\}\(M\_\{\\theta\}\)\>5\\times\\mathcal\{L\}\_\{v\}\(M\_\{ref\}\), or accuracy collapsesFine\-tuning destroys predictive ability; the model outputs near\-constant predictions\. Any separation is a numerical artifact, not memorization\. Signal persists unchanged when labels are shuffled\.Loss Inversionℓ¯atkN​M​\(Mθ\)<ℓ¯atkM​\(Mθ\)\\bar\{\\ell\}^\{NM\}\_\{\\text\{atk\}\}\(M\_\{\\theta\}\)<\\bar\{\\ell\}^\{M\}\_\{\\text\{atk\}\}\(M\_\{\\theta\}\)A pre\-trained structural bias causes non\-members to have lower mean probe loss than members across all context manipulations\. Loss\-based MIA signals are unreliable; distributional metrics \(e\.g\., entropy\) should be checked instead\.Detectable MemorizationAUC​\(Δ​S\)\>0\.5\\text\{AUC\}\(\\Delta S\)\>0\.5on loss or entropyThe fine\-tuned model exposes a calibrated membership signal above the pre\-trained baseline\. The signal weakens materially under shuffled labels, confirming it tracks actual label content\.Context Overfitting\|μgap\|<0\.05\|\\mu\_\{\\text\{gap\}\}\|<0\.05andσgap<0\.2\\sigma\_\{\\text\{gap\}\}<0\.2Fine\-tuning encodes dataset schema or label distribution, not individual records\. Every context manipulation yields the same near\-zero member–non\-member probe loss gap \(stable null\)\.No SignalCatch\-all \(none of the above\)AUC​\(Δ​S\)≤0\.5\\text\{AUC\}\(\\Delta S\)\\leq 0\.5on all metrics with high variance across manipulations \(σgap\\sigma\_\{\\text\{gap\}\}large\)\. The model does not systematically distinguish members from non\-members under any context manipulation\.### A\.1Addressing Common Pitfalls

Tab\.[4](https://arxiv.org/html/2606.31208#A1.T4)summarizes the four common MIA evaluation pitfalls identified by\(Evertzet al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib164)\)and the corresponding countermeasures implemented inIclMem\.

Table 4:Common MIA evaluation pitfalls from\(Evertzet al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib164); Zhanget al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib154)\)andIclMemcorresponding countermeasures\.PitfallIssueCountermeasure inIclMemBase\-rate fallacyMajority\-class prediction is mistaken for a membership signal\.Label randomization forces the model to learn improbable mappings; detectable memorization is the only way to predict the randomized label\.Distributional shiftTrain/test distributions differ, making the attack detect the shift rather than memorization\.Statistical split validation via TVD \(classification\) and the KS test \(regression\) ensures members and non\-members are identically distributed\.Feature contaminationDuplicate rows inflate the attack score by appearing in both member and non\-member sets\.Strict deduplication is enforced before partitioning\.No calibrationLow sample complexity \(easy queries\) is confused with memorization\.Adapting the difficulty calibration idea of\(Watsonet al\.,[2022](https://arxiv.org/html/2606.31208#bib.bib175)\)to our shadow\-model\-free setting, we compute per\-instance delta scoresΔ​S​\(q\)=S​\(q\)−Sr​e​f​\(q\)\\Delta S\(q\)=S\(q\)\-S\_\{ref\}\(q\)using the pre\-trained reference modelMr​e​fM\_\{ref\}; onlyAUC​\(Δ​S\)\>0\.5\\text\{AUC\}\(\\Delta S\)\>0\.5is treated as a detectable signal\.Unverified ground truthThe exact set of records processed during training is unknown, making it impossible to bound the FPR without shadow models\(Zhanget al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib154)\)\.We log every context\-query index pair processed during fine\-tuning\. Only rows provably seen by the model are labeled as members, eliminating label noise from random subsampling or on\-the\-fly data filtering\.
### A\.2Dataset Details

Table[5](https://arxiv.org/html/2606.31208#A1.T5)lists the 10 tasks evaluated in our experiments, all drawn from the CARTE benchmark\(Kimet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib160)\)\. We selected these datasets to cover a heterogeneous mix of real\-world classification and regression domains\. Crucially, all evaluated datasets are strictly excluded from the ConTextTab pre\-training corpus to ensure a valid zero\-shot baseline\.

Table 5:Evaluated datasets with target column and task type, i\.e\., C\(lassification\) and R\(egression\), from CARTE\(Kimet al\.,[2024](https://arxiv.org/html/2606.31208#bib.bib160)\)Table NameTargetTypeanime\_planetRating\_ScoreRbabies\_r\_uspriceRbeer\_ratingsreview\_overallRbikedekhopriceRbikewalepriceRbuy\_buy\_babypriceRcardekhopriceRchocolate\_bar\_ratingsRatingCclear\_corpusBT\_EasinessRcoffee\_ratingsratingCMember/Non\-Member Split Validation\.Tab\.[6](https://arxiv.org/html/2606.31208#A1.T6)reports distributional similarity between the member and non\-member splits for all 10 datasets at bothQ=50Q=50andQ=512Q=512\. For regression, we calculate the Kolmogorov\-Smirnov \(KS\) statistic\(Massey Jr,[1951](https://arxiv.org/html/2606.31208#bib.bib165)\)on the continuous target values, i\.e\., the maximum difference between the empirical cumulative distribution functions, to bound the distance between splits\. For classification, we compute the Total Variation Distance \(TVD\)\(Gibbs and Su,[2002](https://arxiv.org/html/2606.31208#bib.bib166)\)over the label distribution\. All KS statistics are below0\.050\.05\(maximum:0\.0350\.035forbuy\_buy\_babyatQ=512Q=512\)\. Furthermore, all KSpp\-values are above the0\.050\.05significance threshold \(minimum:0\.5300\.530forbuy\_buy\_babyatQ=512Q=512\), meaning we fail to reject the null hypothesis that the member and non\-member splits are drawn from the same distribution\. This confirms that no statistically significant distributional shift exists between our data partitions\. Similarly, TVD values for classification tasks are small \(≤0\.056\\leq 0\.056\), indicating near\-identical label distributions across splits\. We additionally report the Wasserstein distance for completeness\(Mareket al\.,[2025](https://arxiv.org/html/2606.31208#bib.bib191)\); note that this metric is scale\-dependent and therefore not directly comparable across datasets\.

Table 6:Member vs\. non\-member distributional similarity per dataset and query size\. Regression tasks use a two\-sample KS test; classification tasks use TVD over the label distribution\. All KS stat values are<0\.05<0\.05and all KSpp\-values are\>0\.05\>0\.05, confirming no statistically significant distributional shift between member and non\-member splits\. TVD values for classification tasks are similarly small \(≤0\.056\\leq 0\.056\), indicating near\-identical label distributions across splits\. We additionally report the Wasserstein distance for completeness, but note that this metric is scale\-dependent and not directly comparable across datasets with different target ranges\.DatasetTaskQQKS statKSppWTVDQ=512Q=512anime\_planetregr\.5120\.0290\.6110\.025—babies\_r\_usregr\.5120\.0210\.9680\.030—beer\_ratingsregr\.5120\.0210\.9750\.021—bikedekhoregr\.5120\.0250\.8890\.014—bikewaleregr\.5120\.0330\.5950\.011—buy\_buy\_babyregr\.5120\.0350\.5300\.065—cardekhoregr\.5120\.0280\.7630\.007—clear\_corpusregr\.5120\.0230\.9410\.032—chocolate\_bar\_ratingsclass\.512———0\.044coffee\_ratingsclass\.512———0\.056Q=50Q=50anime\_planetregr\.500\.0260\.9830\.024—babies\_r\_usregr\.500\.0201\.0000\.023—beer\_ratingsregr\.500\.0250\.9930\.016—bikedekhoregr\.500\.0270\.9830\.012—bikewaleregr\.500\.0440\.6220\.013—buy\_buy\_babyregr\.500\.0270\.9840\.041—cardekhoregr\.500\.0320\.9210\.010—clear\_corpusregr\.500\.0270\.9880\.038—chocolate\_bar\_ratingsclass\.50———0\.005coffee\_ratingsclass\.50———0\.043
### A\.3Context Manipulations

The probing stage subjects each multiple\-choice context to a structured suite of 48 manipulationsΠ\\Pi, designed to degrade context utility and expose any reliance on parametric memory\. Tab\.[7](https://arxiv.org/html/2606.31208#A1.T7)enumerates all configurations across 11 categories of manipulation types\. Below we describe the 4 main categories\.

1. 1\.Positional Shuffling:Alters the row index of the true target within the context \(random, first, or last position\)\. LTMs are designed to be permutation\-invariant to row order; a model relying solely on ICL should produce identical predictions regardless of ordering\. However, fine\-tuning on static, fixed\-order context\-query pairs can introduce positional biases if optimization exploits ordering shortcuts\. This manipulation tests whether any membership signal is feature\-encoded in the model’s weights or an artifact of positional over\-indexing\.
2. 2\.Distribution Skewing:Skews the marginal label distribution in the context\.*Skew true*over\-represents the true target \(amplification factor∈\{2,3,5\}\\in\\\{2,3,5\\\}\);*Skew incorrect*over\-representsk∈\{1,2,3\}k\\in\\\{1,2,3\\\}incorrect targets \(factor∈\{2,3,5\}\\in\\\{2,3,5\\\}\);*Mixed skewing*adjusts both simultaneously\.*Uniform and random scaling*amplify all candidate frequencies uniformly or randomly \(factor∈\{2,5,10\}\\in\\\{2,5,10\\\}\)\.
3. 3\.Feature Alteration:*Feature reduction*drops 2 valid feature columns to obscure pattern recognition;*Distractor injection*appends 1–2 entirely random columns to introduce spurious signal\. Both test whether the memorization signal survives changes to feature space\.
4. 4\.Combined Stress Tests:2\-way combinations pair positional shuffling with a single distributional or structural manipulation\. 3\-way stress tests simultaneously apply spatial, distributional, and structural degradation\.

Table 7:The complete suite of 48 adversarial context manipulations utilized to evaluate predictive robustness\.Perturbation CategoryVarying ParametersCountDescriptionBaselineN/A1Unperturbed, zero\-information multiple\-choice context\.Positional ShufflingPosition∈\{Random,First,Last\}\\in\\\{\\text\{Random\},\\text\{First\},\\text\{Last\}\\\}3Alters the spatial index of the true target row within the context sequence\.Feature ReductionColumns removed=2=21Drops valid feature columns to obscure pattern recognition\.Distractor InjectionNoisy columns∈\{1,2\}\\in\\\{1,2\\\}2Appends entirely random/noisy features to the context\.Uniform ScalingReplication factor∈\{2,5,10\}\\in\\\{2,5,10\\\}3Uniformly amplifies the frequency of all candidate labels in the context\.Random ScalingReplication factor∈\{2,5,10\}\\in\\\{2,5,10\\\}3Randomly alters the distribution frequencies of all candidate labels in the context\.Skew True TargetAmplification factor∈\{2,3,5\}\\in\\\{2,3,5\\\}3Over\-represents the true target label\.Skew Incorrect TargetsTargets skewedk∈\{1,2,3\}k\\in\\\{1,2,3\\\} Factor∈\{2,3,5\}\\in\\\{2,3,5\\\}9Over\-representskkincorrect candidate labels\.Mixed SkewingTrue factor∈\{2,3,5\}\\in\\\{2,3,5\\\}Incorrect factor∈\{2,3,5\}\\in\\\{2,3,5\\\}7Simultaneously skews both the true target and incorrect targets\.2\-Way CombinationsShuffle \+ \{Uniform, Skewed, Random, Reduce\}9Combines positional shuffling with a single distributional or structural manipulation\.3\-Way Stress TestsShuffle \+ Skew/Uniform/Random \+ Feature Reduction7Stress tests simultaneously applying spatial, distributional, and structural degradation\.Total Configurations48

The count column in Tab\.[7](https://arxiv.org/html/2606.31208#A1.T7)specifies the number of distinct parameter configurations per category; the combinatorial categories \(2\-way, 3\-way combinations\) compose multiple single\-axis manipulations to stress\-test signal robustness simultaneously\. Non\-members rely entirely on the context for generalization and should exhibit clear loss spikes under strong degradation\. Members, whose target mapping is encoded in the model’s weights, should maintain a stable probe loss across all manipulations\.

Perturbation Effectiveness\.We evaluate perturbation effectiveness on the detectable memorization configurations, measuring the mean loss\-basedAUC​\(Δ​S\)\\text\{AUC\}\(\\Delta S\)per perturbation\.*Distribution skewing*is the most effective single perturbation: amplifying the true target label by2×2\\timesyields AUC=0\.557=0\.557\(Δ\+0\.015\\Delta\{\+\}0\.015\), and simultaneously amplifying all labels by5×5\\timesyields the highest observed AUC of0\.5590\.559\(Δ\+0\.017\\Delta\{\+\}0\.017\)\. This aligns with the design rationale: a memorized member resists context pressure because the correct label is encoded in its weights, whereas a non\-member simply follows the over\-represented label\. Also, as expected,*Positional shuffling*is entirely ineffective \(AUC≈0\.543\\approx 0\.543,Δ≈0\\Delta\\approx 0\), confirming that memorization is stored as weight\-level feature\-label associations rather than positional heuristics, and that the model is indeed permutation\-invariant to row order\. Feature reduction and distractor injection produce negligible gains \(AUC0\.5430\.543–0\.5460\.546\)\. Combining positional shuffling with distribution skewing \(shuffle\_and\_skewed\_2: AUC=0\.557=0\.557\) does not improve over skewing alone, and heavily over\-representing incorrect labels \(5×5\\times\) slightly suppresses the signal \(AUC=0\.539=0\.539\), likely by reducing prediction confidence across all labels including the true one\. No individual perturbation amplifies the signal by more thanΔ​AUC≈0\.017\\Delta\\text\{AUC\}\\approx 0\.017; the practical value of the full perturbation suite lies in aggregating these stable but moderate individual signals into a more robust membership discriminator\. Future work could explore more aggressive perturbations \(e\.g\., based on label cardinality or feature importance\) to further amplify signal\.

−0\.5\-0\.500\.50\.5111\.51\.5⋅10−2\\cdot 10^\{\-2\}Pos\. ShufflingFeature Alt\.Non\-True SkewUniform/Rnd\.CombinedMixed SkewTrue\-Label SkewbaselineMeanΔ​AUC​\(Δ​S\)\\Delta\\text\{AUC\}\(\\Delta S\)over baseline

Figure 4:MeanΔ​AUC​\(Δ​S\)\\Delta\\text\{AUC\}\(\\Delta S\)relative to the unperturbed baseline \(AUCbaseline=0\.543\\text\{AUC\}\_\{\\text\{baseline\}\}\{=\}0\.543\), per perturbation category averaged over detectable memorization configurations\. True\-label distribution skewing provides the strongest average lift \(Δ\+0\.013\\Delta\{\+\}0\.013; best single perturbation:mixed\_true\_5\_non\_true\_5,Δ\+0\.017\\Delta\{\+\}0\.017\)\. Positional shuffling is indistinguishable from baseline \(Δ≈0\\Delta\{\\approx\}0\)\.
### A\.4The Necessity of Difficulty Calibration

Raw, absolute probing metrics are unreliable for auditing memorization\(Watsonet al\.,[2022](https://arxiv.org/html/2606.31208#bib.bib175)\)\. Pre\-trained models already encode strong semantic priors and structural logic, achieving high prediction confidence on inherently easy datasets without task\-specific fine\-tuning\. Relying on absolute metrics confounds this pre\-existing structural bias with training\-induced memorization\.

Impact on True Positive Rates \(TPR\)\.Across our530530structurally healthy \(non\-collapsed\) configurations, raw loss evaluation flagged241241runs with uncalibratedTPR@​10%≥0\.10\\text\{TPR\}\_\{@10\\%\}\\geq 0\.10\. Difficulty calibration againstMr​e​fM\_\{ref\}revealed that230230of these cases \(95%95\\%\) were purely data artifacts: the pre\-trained model naturally extracted these records prior to fine\-tuning due to inherent sample easiness\. Applying the calibrated margin \(Δ​TPR@​10%≥0\.10\\Delta\\text\{TPR\}\_\{@10\\%\}\\geq 0\.10\) reduces this to1111configurations with detectable memorization\.

Impact on AUC\-ROC\.Global distributional metrics exhibit the same bias\. If a dataset’s features correlate strongly with a target label under the model’s pre\-training priors,Mr​e​fM\_\{ref\}may yield a raw AUC of0\.850\.85through zero\-shot inference alone\. An uncalibrated audit would flag a subsequent fine\-tuned AUC of0\.870\.87as a large signal\. Computing delta scoresΔ​S​\(q\)=S​\(q\)−Sr​e​f​\(q\)\\Delta S\(q\)=S\(q\)\-S\_\{ref\}\(q\)and thenAUC​\(Δ​S\)\\text\{AUC\}\(\\Delta S\)correctly identifies this as a structural artifact: since the fine\-tuning contributes minimal additional discriminative power,AUC​\(Δ​S\)≈0\.5\\text\{AUC\}\(\\Delta S\)\\approx 0\.5, indistinguishable from random guessing\. Instance\-level calibration viaΔ​S​\(q\)\\Delta S\(q\)is therefore necessary to isolate parametric memorization\.

## Appendix BExtended Evaluation Analysis

While Section[5](https://arxiv.org/html/2606.31208#S5)in the main text summarizes the five observed outcomes and their aggregate rates, this appendix provides a deeper analytical breakdown\. We first detail the dominant artifact outcomes \(representational collapse and loss inversion\), which account for the majority of configurations, then analyze the secondary edge\-case outcomes \(context overfitting and zero\-signal runs\), track the temporal evolution of the memorization signal across fine\-tuning epochs, and explore the disconnect between global and localized memorization\.

### B\.1Artifact Outcomes: Collapse and Inversion

0202040406060No Sig\.Ctx\. Ovft\.Mem\.InversionCollapse55\.255\.219\.319\.312125\.55\.58850\.950\.919\.919\.91\.81\.86\.76\.720\.820\.8% of ConfigurationsQ=50Q=50Q=512Q=512

Figure 5:Distribution ofIclMemoutcomes\. Abbreviations are:*Mem*\(orization\),*Ctx\. Ovft\.*: context overfitting;*No Sig*\(nal\)Outcome I: Representational Collapse\.Affecting over half of our configurations \(55\.2%55\.2\\%atQ=50Q=50;50\.9%50\.9\\%atQ=512Q=512\),*representational collapse*is the dominant fine\-tuning artifact\. At aggressive learning rates \(η≥10−2\\eta\\geq 10^\{\-2\}\), LTMs lose their in\-context reasoning capability and output near\-constant predictions\. While this yields high uncalibrated AUCs \(\>0\.7\>0\.7\), difficulty calibration confirms these are pre\-existing dataset biases rather than induced memorization\. We formally flag collapsed models using an empirically derived threshold:ℒF​T\>5×ℒB​a​s​e\\mathcal\{L\}\_\{FT\}\>5\\times\\mathcal\{L\}\_\{Base\}\. Across our 1,128 runs, the loss ratio \(ℒF​T/ℒB​a​s​e\\mathcal\{L\}\_\{FT\}/\\mathcal\{L\}\_\{Base\}\) follows a strict bimodal distribution\. Stable configurations \(η≤10−4\\eta\\leq 10^\{\-4\}\) tightly cluster between1\.01\.0–2\.0×2\.0\\times, whereas aggressive rates cause rapid degradation, pushing median ratios above8\.0×8\.0\\times\(and frequently exceeding10410^\{4\}\)\. This5×5\\timesboundary robustly separates structurally broken models from true parametric memorization\.

Outcome II: Loss Inversion\.In19\.3%19\.3\\%of configurations atQ=50Q=50, there is a further artifact:*loss inversion*, where non\-members yield lower mean probe loss than members after fine\-tuning\. This occurs when training loss is already inverted in the pre\-trained model, or when the collapsed model’s static predictions align better with non\-members\. Our framework addresses this structural bias by evaluating distributional metrics \(i\.e\., entropy and confidence\) which remain discriminative even when loss is inverted\.

### B\.2Secondary Dynamics and Edge Cases

Our methodology isolates two additional outcomes beyond detectable memorization and representational collapse\.

Outcome IV: Context Overfitting\.In 5\.5% of runs atQ=50Q=50, the model learns statistical properties of the dataset schema rather than individual record membership\. To classify this outcome, we definegk=ℓ¯atkM​\(Mθ,πk\)−ℓ¯atkN​M​\(Mθ,πk\)g\_\{k\}=\\bar\{\\ell\}^\{M\}\_\{\\text\{atk\}\}\(M\_\{\\theta\},\\pi\_\{k\}\)\-\\bar\{\\ell\}^\{NM\}\_\{\\text\{atk\}\}\(M\_\{\\theta\},\\pi\_\{k\}\)as the mean member minus non\-member probe loss gap under perturbationπk\\pi\_\{k\}, and letμgap\\mu\_\{\\text\{gap\}\}andσgap\\sigma\_\{\\text\{gap\}\}denote its mean and standard deviation across all 48 manipulations inΠ\\Pi\. The mean probe loss gap \(μgap\\mu\_\{\\text\{gap\}\}\) is near zero, butσgap\\sigma\_\{\\text\{gap\}\}is minimal across all 48 context manipulations: the zero is stable on every perturbation, not noisy\. This indicates the model treats members and non\-members identically under all context manipulations, having overfitted to the table format rather than individual rows\.

Outcome V: No Signal\.This outcome captures configurations \(8\.0% atQ=50Q=50; 20\.8% atQ=512Q=512\) whereIclMemproduces no systematic pattern\. Noisy and inconsistent per\-manipulation probe loss gaps \(gkg\_\{k\}\) indicate that fine\-tuning improved task performance without inducing a detectable row\-level membership shift\.

### B\.3Overall memorization \(AUC\) vs confident instance signal \(TPR at low FPR\)

Our calibrated audit detected memorization \(AUC​\(Δ​S\)\>0\.5\\text\{AUC\}\(\\Delta S\)\>0\.5\) in 8 of the 10 evaluated tasks under specific hyperparameter regimes, despite representational collapse and chance predictions being the predominant outcomes\. Tab\.[8](https://arxiv.org/html/2606.31208#A2.T8)reports the peak vulnerability configuration for each of these 8 tasks: learning rate, epoch, context size \(QQ\), and the label randomization condition where the peak signal was detected\. Normal indicates the true\-label condition; Shuffled \(M\) indicates member\-only label scrambling; Shuffled \(All\) indicates full\-dataset scrambling\. We report the calibratedAUC​\(Δ​S\)\\text\{AUC\}\(\\Delta S\)to capture global memorization trends across all records, complemented byTPRat lowFPR\(TPR@​10%\\text\{TPR\}\_\{@10\\%\}andTPR@​1%\\text\{TPR\}\_\{@1\\%\}\) to measure localized signal of individual samples\. We reportA​U​CAUCandT​P​RTPRfor the metric \(loss or entropy\) yielding the maximum signal\.

Table 8:The peak detectability configurations for the 8 CARTE datasets, i\.e\., showing detectable memorization\. The AUC and TPR metrics are computed directly on delta scores \(Δ​S\\Delta S\), where an AUC of 0\.50 represents random guessing\.DatasetLabel ConditionLREpochQPrimary MetricAUCTPR@10%TPR@1%babies\_r\_usNormal10−310^\{\-3\}1512Loss0\.6680\.2050\.008bikewaleShuffled \(M\)10−410^\{\-4\}25050Loss0\.6240\.0000\.000chocolate\_bar\_ratingsShuffled \(M\)10−210^\{\-2\}10050Loss0\.6110\.1800\.020buy\_buy\_babyNormal10−410^\{\-4\}1050Loss0\.6080\.1400\.120cardekhoNormal10−410^\{\-4\}150Loss0\.5960\.1400\.000beer\_ratingsShuffled \(M\)10−410^\{\-4\}50512Loss0\.5860\.4940\.166bikedekhoNormal10−410^\{\-4\}10512Loss0\.5790\.1480\.016coffee\_ratingsNormal10−310^\{\-3\}1050Loss0\.5600\.2000\.000

Global vs\. Localized Memorization Signal\.Several configurations in Tab\.[8](https://arxiv.org/html/2606.31208#A2.T8)yield a positive global memorization signal \(AUC​\(Δ​S\)\>0\.5\\text\{AUC\}\(\\Delta S\)\>0\.5\) while simultaneously exhibiting low TPR at strict FPR thresholds\. Fine\-tuning can induce a global distributional shift that separates members from non\-members on average \(AUC\) without concentrating that signal at the individual\-record level:Mr​e​fM\_\{ref\}may achieve higher extreme\-tail confidence on a subset of records purely through zero\-shot structural bias\. A detectable global signal therefore does not imply a strong localized one; row\-level memorization is confined to a smaller subset of configurations under specific hyperparameter regimes\.

### B\.4Data Characteristic Analysis

Fig\.[6](https://arxiv.org/html/2606.31208#A2.F6)plots all 10 evaluated datasets in the target\-cardinality–majority\-class\-frequency space, colored by memorization outcome category\. Tab\.[9](https://arxiv.org/html/2606.31208#A2.T9)reports the underlying statistics and per\-QQdetectable memorization run counts\.

210502005002k10k00\.20\.20\.40\.40\.60\.60\.80\.811beer\_ratingsanime\_planetcardekhoNumber of Target ClasseskkMajority Class Freq\.α\\alphaBothQQQ=50Q=50onlyQ=512Q=512onlyNo Signal

Figure 6:Memorization signal category for each of the 10 evaluated CARTE datasets as a function of target cardinalitykkand majority class frequencyα\\alpha\. The two points atk=2k\{=\}2representchocolate\_bar\_ratingsandcoffee\_ratings; the two points neark=300k\{=\}300representbikedekho\(Q=512 only,k=303k=303\) andbuy\_buy\_baby\(Q=50 only,k=313k=313\)\.Table 9:Dataset statistics and detectable memorization run counts for all 10 evaluated CARTE tasks, sorted by target cardinalitykk\.nn: dataset size;α\\alpha: majority class frequency; Conf@50 / Conf@512: number of configurations yielding detectable memorization atQ=50Q\{=\}50andQ=512Q\{=\}512, respectively\.Datasetnnkkα\\alphaConf@50Conf@512chocolate\_bar2\.6k20\.571310coffee\_ratings2\.1k20\.59810babies\_r\_us5\.1k2020\.08613bikedekho4\.8k3030\.02502buy\_buy\_baby10\.7k3130\.071190bikewale9\.0k3630\.068141beer\_ratings3\.2k23250\.03003anime\_planet14\.4k35160\.00100clear\_corpus4\.7k47240\.00000cardekho37\.8k68650\.02111Feature Composition\.CARTE datasets are mixed with both categorical and numerical features:beer\_ratingshas 14 of 19 features as numeric, i\.e\., the highest ratio among all memorized datasets, producing near\-unique feature vectors that ICL cannot generalize from, contributing to its parametric memorization atQ=512Q=512\. In contrast,clear\_corpus\(15/26 numeric\) has similar richness but its continuous readability targets are feature\-predictable, suppressing memorization\. For predominantly categorical datasets, feature type does not discriminate memorized from non\-memorized tasks; target cardinality and class concentration remain the primary factors\.

## Appendix CExhaustive Hyperparameter Results

Tabs\.[10](https://arxiv.org/html/2606.31208#A3.T10)and[11](https://arxiv.org/html/2606.31208#A3.T11)report outcome metrics forQ=50Q=50andQ=512Q=512respectively, covering all 8 CARTE datasets and all evaluated learning rates\. For each query size, we report the epoch yielding the highest signal of the reported outcome per learning rateη\\eta, selected independently for eachQQ\.

Table 10:Exhaustive evaluation of 8 CARTE datasets underIclMemwithQ=50Q=50context queries\. For each dataset, we track metrics across all four learning rates, selecting the peak\-signal epoch independently forQ=50Q=50\.Δ​ℒv\\Delta\\mathcal\{L\}\_\{v\}: vanilla loss deltaℒv​\(Mθ\)−ℒv​\(Mr​e​f\)\\mathcal\{L\}\_\{v\}\(M\_\{\\theta\}\)\-\\mathcal\{L\}\_\{v\}\(M\_\{ref\}\)for members \(IN\) and non\-members \(OUT\)\.N: normal label condition;S: member\-shuffled condition\.AUC​\(Δ​S\)\\text\{AUC\}\(\\Delta S\)and TPR are computed on instance\-level delta scoresΔ​S​\(q\)=S​\(q\)−Sref​\(q\)\\Delta S\(q\)=S\(q\)\-S\_\{\\text\{ref\}\}\(q\); bold AUC = detectable memorization \(AUC​\(Δ​S\)\>0\.5\\text\{AUC\}\(\\Delta S\)\>0\.5\)\.Pert\.: meanμgap\\mu\_\{\\text\{gap\}\}and std\.σgap\\sigma\_\{\\text\{gap\}\}of the per\-manipulation member–non\-member probe loss gapgkg\_\{k\}across allπk∈Π\\pi\_\{k\}\\in\\Pi\.Outcome:Mem\.= Detectable Memorization, Coll\. = Collapse, Inv\. = Inversion, Ctx\.Ov\. = Context Overfitting, No Sig\. = No Signal\.Δ​ℒv\\Delta\\mathcal\{L\}\_\{v\}AUC​\(Δ​S\)\\text\{AUC\}\(\\Delta S\)Pert\.TPRDatasetLREp\.INOUTNSμgap\\mu\_\{\\text\{gap\}\}σgap\\sigma\_\{\\text\{gap\}\}@10%@1%Outcomebabies\_r\_us10−110^\{\-1\}101\.9⋅1071\.9\{\\cdot\}10^\{7\}1\.4⋅1071\.4\{\\cdot\}10^\{7\}1\.0001\.000−3\.7⋅105\-3\.7\{\\cdot\}10^\{5\}9\.2⋅1049\.2\{\\cdot\}10^\{4\}1\.001\.00Coll\.10−210^\{\-2\}509\.1⋅1049\.1\{\\cdot\}10^\{4\}6\.8⋅1046\.8\{\\cdot\}10^\{4\}0\.9970\.692−1\.9⋅103\-1\.9\{\\cdot\}10^\{3\}573\.40\.980\.98Coll\.10−310^\{\-3\}10\.5020\.5780\.4620\.464\-0\.1480\.7370\.040No Sig\.10−410^\{\-4\}50\-0\.184\-0\.0340\.665\\mathbf\{0\.665\}0\.456\-0\.1610\.7240\.020Mem\.bikewale10−110^\{\-1\}12\.8⋅1042\.8\{\\cdot\}10^\{4\}2\.2⋅1042\.2\{\\cdot\}10^\{4\}0\.3740\.48120\.4778\.790\.020Coll\.10−210^\{\-2\}507\.4⋅1047\.4\{\\cdot\}10^\{4\}5\.8⋅1045\.8\{\\cdot\}10^\{4\}0\.3500\.59755\.54168\.70\.020Coll\.10−310^\{\-3\}10\.7340\.9590\.5180\.486\-0\.0390\.3330\.080\.04Coll\.10−410^\{\-4\}50\-0\.0520\.0090\.578\\mathbf\{0\.578\}0\.563\\mathbf\{0\.563\}\-0\.0170\.15600Mem\.chocolate\_bar\_ratings10−110^\{\-1\}50\-0\.0240\.1650\.585\\mathbf\{0\.585\}0\.594\\mathbf\{0\.594\}\-0\.0070\.0630\.160Mem\.10−210^\{\-2\}50\-0\.0100\.1450\.604\\mathbf\{0\.604\}0\.605\\mathbf\{0\.605\}0\.0040\.0410\.160Mem\.10−310^\{\-3\}100\.0210\.1620\.609\\mathbf\{0\.609\}0\.606\\mathbf\{0\.606\}0\.0090\.0850\.180\.02Mem\.10−410^\{\-4\}100\.0990\.0390\.4660\.589\\mathbf\{0\.589\}0\.0540\.3100\.100\.02Mem\.buy\_buy\_baby10−110^\{\-1\}19\.2⋅1039\.2\{\\cdot\}10^\{3\}5\.9⋅1035\.9\{\\cdot\}10^\{3\}0\.9981\.000\-761\.1313\.91\.000\.96Coll\.10−210^\{\-2\}103\.8⋅1043\.8\{\\cdot\}10^\{4\}2\.4⋅1042\.4\{\\cdot\}10^\{4\}1\.0001\.000−2\.8⋅103\-2\.8\{\\cdot\}10^\{3\}722\.21\.001\.00Coll\.10−310^\{\-3\}10\.4520\.2690\.553\\mathbf\{0\.553\}0\.563\\mathbf\{0\.563\}\-0\.3071\.8710\.100Mem\.10−410^\{\-4\}10\-0\.060\-0\.0110\.608\\mathbf\{0\.608\}0\.577\\mathbf\{0\.577\}\-0\.2131\.7910\.140\.12Mem\.cardekho10−110^\{\-1\}1356\.1537\.50\.3510\.0410\.9022\.0840\.040Coll\.10−210^\{\-2\}17\.70512\.750\.5460\.540\-0\.0260\.3240\.240\.06Coll\.10−310^\{\-3\}10\.8080\.8640\.5580\.5560\.0090\.0700\.240\.06Coll\.10−410^\{\-4\}10\.0100\.0280\.596\\mathbf\{0\.596\}0\.4970\.0110\.0670\.140Mem\.beer\_ratings10−110^\{\-1\}13\.3⋅1043\.3\{\\cdot\}10^\{4\}2\.8⋅1042\.8\{\\cdot\}10^\{4\}0\.0190\.294286\.4119\.200Coll\.10−210^\{\-2\}15\.2494\.7650\.4800\.511\-0\.0031\.5580\.080\.04Coll\.10−310^\{\-3\}11\.4621\.2650\.4810\.495\-0\.0070\.4600\.100No Sig\.10−410^\{\-4\}10\-0\.1980\.1210\.5070\.509\-0\.0390\.5300\.080\.02No Sig\.bikedekho10−110^\{\-1\}13\.7⋅1033\.7\{\\cdot\}10^\{3\}3\.0⋅1033\.0\{\\cdot\}10^\{3\}——57\.8316\.0700Coll\.10−210^\{\-2\}12\.9861\.7050\.4280\.4360\.0570\.3610\.080Coll\.10−310^\{\-3\}14\.2623\.4230\.5370\.532\-0\.0350\.2360\.100\.06Ctx\.Ov\.10−410^\{\-4\}10\-0\.0050\.0180\.5420\.530\-0\.0050\.1170\.140\.06Ctx\.Ov\.coffee\_ratings10−110^\{\-1\}109\.3759\.3350\.4980\.5190\.1132\.7170\.080Coll\.10−210^\{\-2\}500\.3070\.2670\.5280\.5360\.0040\.1060\.080\.02Inv\.10−310^\{\-3\}100\.1770\.1370\.560\\mathbf\{0\.560\}0\.508\-0\.0060\.1530\.200Mem\.10−410^\{\-4\}500\.1820\.2920\.5200\.5290\.0110\.1730\.080\.02Inv\.Table 11:Exhaustive evaluation of 8 CARTE datasets underIclMemwithQ=512Q=512context queries\. For each dataset, we track metrics across all four learning rates, selecting the peak\-signal epoch independently forQ=512Q=512\.Δ​ℒv\\Delta\\mathcal\{L\}\_\{v\}: vanilla loss deltaℒv​\(Mθ\)−ℒv​\(Mr​e​f\)\\mathcal\{L\}\_\{v\}\(M\_\{\\theta\}\)\-\\mathcal\{L\}\_\{v\}\(M\_\{ref\}\)for members \(IN\) and non\-members \(OUT\)\.N: normal label condition;S: member\-shuffled condition\.AUC​\(Δ​S\)\\text\{AUC\}\(\\Delta S\)and TPR are computed on instance\-level delta scoresΔ​S​\(q\)=S​\(q\)−Sref​\(q\)\\Delta S\(q\)=S\(q\)\-S\_\{\\text\{ref\}\}\(q\); bold AUC = detectable memorization \(AUC​\(Δ​S\)\>0\.5\\text\{AUC\}\(\\Delta S\)\>0\.5\)\.Pert\.: meanμgap\\mu\_\{\\text\{gap\}\}and std\.σgap\\sigma\_\{\\text\{gap\}\}of the per\-manipulation member–non\-member probe loss gapgkg\_\{k\}across allπk∈Π\\pi\_\{k\}\\in\\Pi\.Outcome:Mem\.= Detectable Memorization, Coll\. = Collapse, Inv\. = Inversion, Ctx\.Ov\. = Context Overfitting, No Sig\. = No Signal\.Δ​ℒv\\Delta\\mathcal\{L\}\_\{v\}AUC​\(Δ​S\)\\text\{AUC\}\(\\Delta S\)Pert\.TPRDatasetLREp\.INOUTNSμgap\\mu\_\{\\text\{gap\}\}σgap\\sigma\_\{\\text\{gap\}\}@10%@1%Outcomebabies\_r\_us10−110^\{\-1\}1206\.1171\.30\.276—18\.0931\.070\.020Coll\.10−210^\{\-2\}10\.7110\.7420\.4870\.463\-0\.0751\.2340\.120\.01Coll\.10−310^\{\-3\}10\.5690\.5350\.668\\mathbf\{0\.668\}—\-0\.1021\.2850\.210\.01Mem\.10−410^\{\-4\}10\-0\.0330\.0040\.608\\mathbf\{0\.608\}0\.482\-0\.0851\.2890\.020Mem\.bikewale10−110^\{\-1\}13\.0⋅1043\.0\{\\cdot\}10^\{4\}2\.7⋅1042\.7\{\\cdot\}10^\{4\}0\.1930\.12169\.1483\.080\.000Coll\.10−210^\{\-2\}149\.0442\.880\.4850\.4660\.0842\.6580\.090\.01Coll\.10−310^\{\-3\}10\.4080\.4120\.582\\mathbf\{0\.582\}—\-0\.0090\.2640\.270\.12Mem\.10−410^\{\-4\}10\-0\.0210\.0080\.5360\.521\-0\.0090\.2640\.110\.01No Sig\.chocolate\_bar\_ratings10−110^\{\-1\}502\.6771\.9560\.4960\.5110\.0081\.1220\.090\.01No Sig\.10−210^\{\-2\}500\.0670\.0250\.5140\.5160\.0000\.0560\.100\.02Inv\.10−310^\{\-3\}100\.0920\.2000\.512—\-0\.0010\.1750\.100\.02Ctx\.Ov\.10−410^\{\-4\}100\.009\-0\.0120\.5160\.510\-0\.0000\.0550\.090\.01Ctx\.Ov\.buy\_buy\_baby10−110^\{\-1\}11\.5⋅1041\.5\{\\cdot\}10^\{4\}1\.7⋅1041\.7\{\\cdot\}10^\{4\}——2\.5⋅1032\.5\{\\cdot\}10^\{3\}686\.500Coll\.10−210^\{\-2\}119\.7827\.810\.4310\.4246\.07935\.420\.050Coll\.10−310^\{\-3\}500\.7640\.7410\.513—\-0\.0692\.8600\.140\.03Coll\.10−410^\{\-4\}50\-0\.024\-0\.0020\.4610\.521\-0\.0022\.3260\.110\.02No Sig\.cardekho10−110^\{\-1\}105\.7⋅1075\.7\{\\cdot\}10^\{7\}6\.0⋅1076\.0\{\\cdot\}10^\{7\}0\.4640\.5361\.2⋅1031\.2\{\\cdot\}10^\{3\}1\.5⋅1041\.5\{\\cdot\}10^\{4\}0\.080\.02Coll\.10−210^\{\-2\}102\.0⋅1032\.0\{\\cdot\}10^\{3\}2\.1⋅1032\.1\{\\cdot\}10^\{3\}0\.5520\.492\-0\.4554\.2240\.110\.02Coll\.10−310^\{\-3\}50250\.3261\.80\.489—0\.0211\.4730\.100\.02Coll\.10−410^\{\-4\}500\.0430\.0560\.560\\mathbf\{0\.560\}0\.4890\.0010\.0660\.030\.00Mem\.beer\_ratings10−110^\{\-1\}102\.3⋅1072\.3\{\\cdot\}10^\{7\}2\.2⋅1072\.2\{\\cdot\}10^\{7\}0\.9990\.909−1\.9⋅105\-1\.9\{\\cdot\}10^\{5\}5\.1⋅1045\.1\{\\cdot\}10^\{4\}1\.001\.00Coll\.10−210^\{\-2\}502\.6⋅1032\.6\{\\cdot\}10^\{3\}2\.5⋅1032\.5\{\\cdot\}10^\{3\}0\.7920\.996\-19\.6826\.750\.240\.04Coll\.10−310^\{\-3\}10\.4920\.4700\.502—\-0\.0340\.6190\.080\.01No Sig\.10−410^\{\-4\}50\-0\.0090\.0260\.4970\.586\\mathbf\{0\.586\}\-0\.0330\.5780\.490\.17Mem\.bikedekho10−110^\{\-1\}11\.0⋅1041\.0\{\\cdot\}10^\{4\}9\.9⋅1039\.9\{\\cdot\}10^\{3\}——218\.232\.4800Coll\.10−210^\{\-2\}15\.1594\.6310\.4420\.4730\.1010\.6430\.070\.01Coll\.10−310^\{\-3\}11\.0291\.0360\.473—0\.0130\.2280\.130\.02Coll\.10−410^\{\-4\}10\-0\.0250\.0070\.579\\mathbf\{0\.579\}0\.4970\.0060\.1900\.150\.02Mem\.coffee\_ratings10−110^\{\-1\}500\.2840\.2460\.5190\.517\-0\.0040\.1560\.090\.00Ctx\.Ov\.10−210^\{\-2\}500\.2630\.2230\.5190\.522\-0\.0030\.1420\.090\.00Ctx\.Ov\.10−310^\{\-3\}100\.1780\.1340\.521—\-0\.0020\.0740\.100\.00Ctx\.Ov\.10−410^\{\-4\}500\.2750\.1280\.5220\.525\-0\.0010\.0790\.100\.01Ctx\.Ov\.
## Appendix DMemorization under Realistic Training Regimes

Table 12:Peak detectable memorization atQ=512Q\{=\}512, for datasets with signal in at least one configuration\.*Single\-Task Fixed Pairs*: one model fine\-tuned per dataset \(peak across allη\\eta;⋆η=10−3\\eta=10^\{\-3\}, othersη=10−4\\eta=10^\{\-4\}\)\.*Multi\-Task Fixed Pairs*: one model trained jointly on all 10 CARTE tasks with fixed context\-query pairs \(η=10−4\\eta=10^\{\-4\}\)\.*Multi\-Task Random Pairs*: same joint setup with a fresh random context\-query sample at each step \(η=10−4\\eta=10^\{\-4\}\)\.Bold: highest AUC per row\.Single\-Task Fixed PairsMulti\-Task Fixed PairsMulti\-Task Random PairsDatasetEpAUCTPR@​10%\\text\{TPR\}\_\{@10\\%\}TPR@​1%\\text\{TPR\}\_\{@1\\%\}EpAUCTPR@​10%\\text\{TPR\}\_\{@10\\%\}TPR@​1%\\text\{TPR\}\_\{@1\\%\}EpAUCTPR@​10%\\text\{TPR\}\_\{@10\\%\}TPR@​1%\\text\{TPR\}\_\{@1\\%\}babies\_r\_us1⋆0\.6680\.2050\.0081000\.6180\.0080\.000—beer\_ratings500\.5860\.4940\.166——bikedekho100\.5790\.1480\.01610\.5580\.1900\.027—bikewale1⋆0\.5820\.2660\.123—100\.5750\.1000\.018cardekho500\.5600\.0290\.002——Memorization Signal under Realistic Regimes \(Q=512Q\{=\}512\)\.Tab\.[12](https://arxiv.org/html/2606.31208#A4.T12)shows how memorization degrades as the training regime progressively approaches practical conditions\. Under single\-task fine\-tuning on fixed context\-query pairs \(worst case\), 5 of 10 tasks are detectable\. By fine\-tuning a single model jointly on all 10 tasks with fixed context\-query pairs, i\.e\., 10 gradient steps per epoch, the signal appears only for 2 tasks \(babies\_r\_us: AUC 0\.668→\\to0\.618;bikedekho: AUC 0\.579→\\to0\.558\)\. We intentionally fixed the context\-query pairs across epochs, however, in practice the pairs are randomly sampled\. With random context\-query sampling and fine\-tuning on all 10 tasks, the signal collapses to a single task:bikewale\(AUC 0\.575\)\. Critically, this residual signal appears only at Epoch 10, already beyond the 2–5 epoch budget of real LTMs pre\-training\([Spinaciet al\.,](https://arxiv.org/html/2606.31208#bib.bib145)\); within that practical budget, no task would be detectable\. The pattern confirms that memorization underIclMemrequires two structural conditions absent from practical pre\-training: \(i\) per\-task gradient specialization and \(ii\) repeated exposure to fixed context\-query pairs\.