Fewer Heads, Fewer Memories: Pruning Redundant Attention Heads Suppresses Verbatim Memorisation

# Fewer Heads, Fewer Memories: Pruning Redundant Attention Heads Suppresses Verbatim Memorisation
Source: [https://arxiv.org/html/2610.40974](https://arxiv.org/html/2610.40974)
Riza Batista\-NavarroAffiliation:Senior Lecturer in Text MiningAffiliation:University of Manchester

###### Abstract

Large language models have been shown to memorise and reproduce verbatim sequences from their training data, raising concerns about privacy leakage, intellectual property exposure, and the limits of generalisation\. Recent theoretical work suggests that attention heads may contribute to this behaviour by functioning as associative memory mechanisms, with memorisation capacity scaling with the number of heads\. At the same time, empirical work has shown that many attention heads are redundant and can be pruned with limited impact on downstream performance\. Despite these parallel findings, the effect of pruning redundant heads on memorised\-sequence reproduction remains unexplored\. In this work, we demonstrate that zeroing out a substantial fraction of attention heads prior to inference can significantly suppress verbatim memorisation while preserving, or even improving, language modelling performance on the benchmark used to identify redundant heads\. Evaluating GPT\-2\-XL\(1\.5B\) on known memorised sequences, we find that pruning 729 task\-redundant heads reduces the memorisation count by 91\.5%, while improving WikiText\-103 perplexity by 0\.92\. These findings suggest that attention\-head redundancy can be exploited to mitigate memorisation at inference time, without retraining or modifying the training data\.

## 1Introduction

Large language models have been shown to memorise and reproduce verbatim sequences from their training data, raising concerns about privacy leakage, intellectual property exposure, and the limits of generalisation\([Carlini et al\., 2021](https://arxiv.org/html/2610.40974#bib.bib5);[Carlini et al\., 2022](https://arxiv.org/html/2610.40974#bib.bib6)\)\. Existing mitigations, such as deduplication, differential privacy, and unlearning\([Kandpal et al\., 2022](https://arxiv.org/html/2610.40974#bib.bib7);[Yu et al\., 2021](https://arxiv.org/html/2610.40974#bib.bib8);[Si et al\., 2023](https://arxiv.org/html/2610.40974#bib.bib9)\), either intervene at training time or require identifying specific memorised content, making them difficult to apply to deployed models\.

Prior work has shown that MLP layers store substantial factual and memorised knowledge[Meng et al\. \(2022\)](https://arxiv.org/html/2610.40974#bib.bib16);[Nanda et al\. \(2023\)](https://arxiv.org/html/2610.40974#bib.bib17), while the role of attention remains comparatively underexplored\. Recent theoretical results establish that multi\-head attention can function as an associative memory, with memorisation capacity scaling in the number of heads\([Mahdavi et al\., 2024](https://arxiv.org/html/2610.40974#bib.bib1);[Dana et al\., 2024](https://arxiv.org/html/2610.40974#bib.bib2)\)\. This capacity arises from softmax saturation: when attention weights concentrate sharply on a small set of positions, the corresponding value vectors dominate the head’s output, enabling precise token retrieval rather than diffuse contextual aggregation\. Independently, empirical work has consistently shown that a substantial fraction of attention heads can be zeroed out at inference time without degrading downstream task performance\([Michel et al\., 2019](https://arxiv.org/html/2610.40974#bib.bib3);[Voita et al\., 2019](https://arxiv.org/html/2610.40974#bib.bib4)\)\. The implications of this redundancy for memorisation, however, remain unexplored\. Our contributions are as follows:

- •We demonstrate that inference\-time pruning of task\-redundant attention heads substantially reduces memorised\-sequence reproduction while preserving language modelling performance on the benchmark used to identify redundant heads\.
- •We find that memorised sequences induce sharper softmax saturation and stronger cross\-head agreement, creating a coordinated signal toward the memorised continuation\.
- •We show that pruning task\-redundant heads weakens this collective signal by removing part of their contribution from the residual stream, offering a mechanistic explanation for the observed reduction in memorisation\.

## 2Methodology

### 2\.1Verbatim Memorisation

Following prior work, we define memorisation usingkk\-extractability\. A sequencessis considered memorised if, when prompted with its firstkktokens, the model reproduces the nextkktokens verbatim under greedy decoding\([Biderman et al\., 2023a](https://arxiv.org/html/2610.40974#bib.bib12)\)\. We selectk=16k=16as the shortest prefix length to capture short\-span memorisation, whilek=32k=32andk=50k=50follow the thresholds used in prior extractability studies\([Carlini et al\., 2021](https://arxiv.org/html/2610.40974#bib.bib5);[Biderman et al\., 2023a](https://arxiv.org/html/2610.40974#bib.bib12)\)\.

### 2\.2Models and Datasets

We conduct experiments on Pythia\-2\.8B\-Deduped, Pythia\-6\.9B\-Deduped, and GPT\-2\-XL \(1\.5B\)\([Biderman et al\., 2023b](https://arxiv.org/html/2610.40974#bib.bib10);[Radford et al\., 2019](https://arxiv.org/html/2610.40974#bib.bib11)\), chosen for their open weights, documented pretraining corpora, and established use in memorisation research\. Together, they span a range of scales and architectures, which is relevant given that memorisation rates increase with model capacity\([Carlini et al\., 2022](https://arxiv.org/html/2610.40974#bib.bib6)\)\.

For each model, memorised and non\-memorised sequences are drawn from its corresponding pretraining corpus: Pile\-Deduped for the Pythia models and OpenWebText for GPT\-2\-XL\([Gao et al\., 2020](https://arxiv.org/html/2610.40974#bib.bib13);[Radford et al\., 2019](https://arxiv.org/html/2610.40974#bib.bib11)\)\. For the Pythia models, we sample 20,000 memorised sequences per model from the public list of memorised sequences\([Biderman et al\., 2023a](https://arxiv.org/html/2610.40974#bib.bib12)\)\. For GPT\-2\-XL, we construct an equivalent set by applying the criterion from Section 2\.1 to every OpenWebText sequence, yielding between 4,800 and 5,590 unique memorised sequences depending onkk\. In all cases, memorisation count is measured as the number of sequences reproduced verbatim under greedy decoding, before and after head pruning\. For each model, we additionally construct a matched non\-memorised set by sampling an equal number of sequences whose greedy continuation does not exactly match the target continuation\.

Performance is evaluated on WikiText\-103 and LAMBADA, standard benchmarks for language modelling and next\-token prediction\([Merity and others, 2016](https://arxiv.org/html/2610.40974#bib.bib14);[Paperno et al\., 2016](https://arxiv.org/html/2610.40974#bib.bib15)\); we report perplexity on both and last\-word prediction accuracy on LAMBADA\. For gate optimisation \(Section[2\.3](https://arxiv.org/html/2610.40974#S2.SS3)\), we use a held\-out split of WikiText\-103/LAMBADA, disjoint from the split used for final perplexity and accuracy reporting\.

### 2\.3Identifying Task\-Redundant Heads

We identify task\-redundant attention heads by learning sparse binary gates over head outputs while keeping all model parameters frozen\. For each headhhin layerll, the gated output is

zl,h′=bl,hzl,h,bl,h=𝟏\[σ\(αl,h\)\>0\.5\],z^\{\\prime\}\_\{l,h\}=b\_\{l,h\}z\_\{l,h\},\\qquad b\_\{l,h\}=\\mathbf\{1\}\[\\sigma\(\\alpha\_\{l,h\}\)\>0\.5\],wherezl,hz\_\{l,h\}is the head output,αl,h\\alpha\_\{l,h\}is a learned gate logit, andbl,hb\_\{l,h\}is a binary gate\. Sincebl,hb\_\{l,h\}is a non\-differentiable step function ofαl,h\\alpha\_\{l,h\}, we use a straight\-through estimator\([Bengio et al\., 2013](https://arxiv.org/html/2610.40974#bib.bib19)\): hard gates are used in the forward pass, while gradients are passed through the sigmoid relaxationσ⁡\(αl,h\)\\sigma\(\\alpha\_\{l,h\}\)in the backward pass, allowing standard backpropagation despite the non\-differentiable forward operation\.

ℒ=max⁡\(0,ℒgated−ℒorig−log⁡r\)\+λ​∑l,hσ⁡\(αl,h\),\\mathcal\{L\}=\\max\\left\(0,\\mathcal\{L\}\_\{\\mathrm\{gated\}\}\-\\mathcal\{L\}\_\{\\mathrm\{orig\}\}\-\\log r\\right\)\+\\lambda\\sum\_\{l,h\}\\sigma\(\\alpha\_\{l,h\}\),whererris the allowed perplexity ratio andλ\\lambdacontrols sparsity\. Since perplexity is exponential in loss, capping the perplexity ratio atrrcaps the increase in loss atlog⁡r\\log r\. The objective thus discourages pruning configurations that exceed the permitted increase in loss, while encouraging fewer active heads\.

#### Implementation details\.

Gate logitsαl,h\\alpha\_\{l,h\}are initialised to5\.05\.0for all heads, corresponding toσ⁡\(αl,h\)≈0\.99\\sigma\(\\alpha\_\{l,h\}\)\\approx 0\.99, so that all heads begin active and pruning proceeds from the full model\. We setr=1\.00r=1\.00across pruning objectives andλ=10−4\\lambda=10^\{\-4\}, and optimise gate logits with AdamW\([Loshchilov and Hutter, 2017](https://arxiv.org/html/2610.40974#bib.bib20)\)at a learning rate of10−210^\{\-2\}, using batches of 8 sequences sampled with replacement from the held\-out gate\-optimisation split, for a fixed budget of 1,000 steps\. Rather than using the gate configuration from the final training step, we track, at each step, the batch\-level perplexity ratioℒgated/ℒorig\\mathcal\{L\}\_\{\\mathrm\{gated\}\}/\\mathcal\{L\}\_\{\\mathrm\{orig\}\}under the current hard gates, and select the gate configuration achieving the lowest observed ratio over the full training run\. Heads with inactive gates in the selected configuration are zeroed out at inference time\.

### 2\.4Mechanistic Analysis Metrics

To analyse how pruning affects memorisation\-related computation, we compute four diagnostics on both memorised and non\-memorised sequences\. For each attention head, we measure Shannon entropy as:

Hl,h=−1N∑n=1N1Tn∑t=1Tn∑j≤tA\(n\)l,h,t,jlogA\(n\)l,h,t,j,H\_\{l,h\}=\-\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\frac\{1\}\{T\_\{n\}\}\\sum\_\{t=1\}^\{T\_\{n\}\}\\sum\_\{j\\leq t\}A^\{\(n\)\}\_\{l,h,t,j\}\\log A^\{\(n\)\}\_\{l,h,t,j\},whereAl,h,t,jA\_\{l,h,t,j\}is the attention weight from positionttto positionjj, averaged across sequences\. Lower entropy indicates more concentrated attention, while higher entropy indicates more diffuse contextual mixing\.

We measure softmax saturation as the number of attention rows with maximum weight above a threshold of0\.950\.95, aggregated across token positions and sequences\. This captures the extent to which a head places nearly all of its attention mass on a single position while remaining permissive enough to surface meaningful differences in saturation across sequence types\([Mahdavi et al\., 2024](https://arxiv.org/html/2610.40974#bib.bib1)\)\.

To assess whether saturated heads coordinate on the same evidence, we measure cross\-head agreement as the number of times attention rows from different heads share the same maximum\-weight position\. Higher agreement indicates that heads are converging on the same position rather than attending independently\.

Finally, we measure the top\-two logit margin,Δlogit=ℓ\(1\)−ℓ\(2\)\\Delta\_\{\\mathrm\{logit\}\}=\\ell\_\{\(1\)\}\-\\ell\_\{\(2\)\}, averaged across all generated positions\. A larger margin indicates a stronger model preference for a single continuation\. Comparing each diagnostic before and after pruning allows us to test whether head removal weakens the collective signal driving memorised reproduction\.

To support reproducibility, we will release all the source code upon acceptance\. We also acknowledge the use of AI assistance during the preparation of this work for grammar and spell checking, code assistance, and improving clarity and readability\.

## 3Results

### 3\.1Effects of Pruning on Memorisation

LM PerformanceMemorisation CountModelVariantActive HeadsWikiText \(PPL\)k=16k\{=\}16k=32k\{=\}32k=50k\{=\}50GPT\-2\-XL\(1\.5B\)Baseline1200 / 120010\.08559055404800Random471 / 1200146\.82140330910Pruned \(WT\)471 / 12009\.163154731103Pythia\-2\.8BBaseline1024 / 10247\.5620,00020,00020,000Random628 / 102418\.30536543264661Pruned \(WT\)628 / 10247\.32472628754770Pythia\-6\.9BBaseline1024 / 10246\.9020,00020,00020,000Random650 / 102417\.20490547514310Pruned \(WT\)650 / 10246\.73422135004202Table 1:Language modelling performance and memorisation counts \(k\-extractability\) under WikiText\-103\-guided pruning \(WT\), alongside baseline and random\-pruning conditions\.LM PerformanceMemorisation CountModelVariantActive HeadsLAMBADA \(PPL/Acc\.\)k=16k\{=\}16k=32k\{=\}32k=50k\{=\}50GPT\-2\-XL\(1\.5B\)Baseline1200 / 120025\.97/0\.6128559055404800Random642 / 120091\.38/0\.2014140330910Pruned \(LB\)642 / 120025\.09/0\.6148133411332200Pythia\-2\.8BBaseline1024 / 102417\.35/0\.748920,00020,00020,000Random775 / 102433\.24/0\.36177981732611661Pruned \(LB\)775 / 102417\.51/0\.75438094699510414Pythia\-6\.9BBaseline1024 / 102414\.80/0\.781220,00020,00020,000Random780 / 102429\.40/0\.410010302750311320Pruned \(LB\)780 / 102414\.95/0\.77608500722211101Table 2:Language modelling performance and memorisation counts \(k\-extractability\) under LAMBADA\-guided pruning \(LB\), alongside baseline and random\-pruning conditions\.Across all models and benchmarks, Tables[1](https://arxiv.org/html/2610.40974#S3.T1)and[2](https://arxiv.org/html/2610.40974#S3.T2)show that pruning substantially reduces memorised\-sequence reproduction while preserving, and in several cases improving, performance on the benchmark used to identify redundant heads\. We first compare the Baseline and Pruned conditions, leaving Random to be discussed separately below\.

Notably, the sets of heads identified as task\-redundant differ between the WikiText\-103 and LAMBADA objectives, and both differ across models, yet both configurations suppress memorisation substantially in every setting\. This suggests that memorisation is not localised in a single narrow subset of heads, but distributed across the attention mechanism, with different task\-preserving subsets still capable of suppressing it when removed\.

Pruning against the WikiText split \(WT\) consistently improves or preserves WikiText perplexity while removing the larger share of heads and yielding the larger memorisation reduction\. GPT\-2\-XL improves from 10\.08 to 9\.16 while removing∼\\sim61% of heads and cutting memorisation from 5590 to 315 atk=16k\{=\}16, and both Pythia models show the same pattern \(7\.56→\\to7\.32 and 6\.90→\\to6\.73\), withk=32k\{=\}32memorisation falling from 20,000 to 2875 and 3500\.

Pruning against the LAMBADA split \(LB\) shows the complementary pattern: fewer heads are removed, and the memorisation reduction is correspondingly smaller, but LAMBADA performance is preserved or slightly improved \(e\.g\. GPT\-2\-XL: 25\.09/0\.6148 vs\. 25\.97/0\.6128 baseline\)\. Together, these results indicate that memorisation reduction scales with the number of heads pruned, regardless of which benchmark’s redundant heads are targeted\.

This relationship does not imply that removing heads indiscriminately would suppress memorisation just as well\. As the random baseline below shows, the identity of the removed heads matters as much as their number\.

To isolate this, we include a random baseline in which the number of pruned heads is set to the same value as the number of heads pruned using the sparse gates\. We run each model 5 times, each with a different random selection of heads, and report the average language modelling performance and memorisation counts \(rounded up to the nearest integer\)\.

Although random pruning sometimes yields lower memorisation counts than targeted pruning \(e\.g\., 330 vs\. 473 memorised sequences atk=32k\{=\}32for GPT\-2\-XL on WT pruning\), this comes at the cost of severe degradation in language modelling capability\. Under WikiText\-guided pruning, GPT\-2\-XL’s WikiText perplexity rises from 10\.08 to 146\.82 under random pruning, with both Pythia models showing comparable degradation \(7\.56→\\to18\.30 and 6\.90→\\to17\.20\)\. Under LAMBADA\-guided pruning, GPT\-2\-XL’s LAMBADA accuracy falls from 0\.6128 to 0\.2014 under random pruning, with both Pythia models again showing substantial degradation \(0\.7489→\\to0\.3617 and 0\.7812→\\to0\.4100\)\. Since a model with severely impaired predictive ability will trivially fail to reproduce training sequences, memorisation count alone is an insufficient measure of a mitigation strategy’s value\. In contrast, task\-guided pruning achieves a substantially better utility–memorisation trade\-off by reducing extractability while retaining the model’s language modelling ability\.

### 3\.2Attention Saturation

For the mechanistic analysis, we focus on Pythia\-2\.8B and GPT\-2\-XL pruned under the WikiText\-103 objective\. We exclude Pythia\-6\.9B due to computational resource constraints, and restrict to a single benchmark objective for clarity of exposition\.

Table 3:Attention entropy on memorised and non\-memorised sequences, before and after pruning\. Total and Average are computed over active heads only, so that values before and after pruning are directly comparable at the per\-head level\.Δ\\DeltaTotal andΔ\\DeltaAverage denote the memorised value minus the non\-memorised value\.Table 4:Softmax saturation counts on memorised and non\-memorised sequences, before and after WikiText\-guided pruning, alongside the corresponding rate per active head \(Δ\\DeltaCount = Mem\. Count−\-Non\-mem\. Count\)\.Tables[3](https://arxiv.org/html/2610.40974#S3.T3)and[4](https://arxiv.org/html/2610.40974#S3.T4)report attention entropy and softmax saturation normalised by the number of active heads, so values before and after pruning are directly comparable per head\. In both unpruned models, memorised sequences yield lower attention entropy \(Δ\\DeltaAverage of−0\.1325\-0\.1325for Pythia\-2\.8B,−0\.0222\-0\.0222for GPT\-2\-XL\) and more softmax saturation \(ratios of1\.401\.40and1\.761\.76\) than non\-memorised ones, consistent with memorised content eliciting more peaked, more confident attention\.

Pruning narrows this saturation gap\. The memorised\-to\-non\-memorised ratio falls to1\.221\.22\(Pythia\-2\.8B\) and1\.321\.32\(GPT\-2\-XL\), and the absolute gap in counts shrinks by about74%74\\%and85%85\\%\. Saturation drops more on memorised sequences than on non\-memorised ones \(Pythia\-2\.8B:−58\.6%\-58\.6\\%vs\.−52\.5%\-52\.5\\%; GPT\-2\-XL:−72\.7%\-72\.7\\%vs\.−63\.5%\-63\.5\\%\)\. Moreover, the reduction is not explained by head count alone: rates per active head fall by32\.5%32\.5\\%and30\.4%30\.4\\%on memorised text, versus22\.5%22\.5\\%and7\.0%7\.0\\%on non\-memorised text, with the contrast most pronounced in GPT\-2\-XL\.

The surviving heads also attend more diffusely\. Mean per\-head entropy on memorised text rises from about2\.082\.08to3\.653\.65in Pythia\-2\.8B and from about1\.271\.27to3\.533\.53in GPT\-2\-XL\. Since our pruning objective does not use entropy, this cannot be attributed to the removal of low\-entropy heads\. A more plausible explanation is a ripple effect: removing heads alters the representations received by later layers, so the remaining heads attend less sharply and saturate less in general\. We do not test this here and leave it to future work\.

Consequently, the entropy gap attenuates rather than disappears\. In Pythia\-2\.8B it shrinks by about68%68\\%\(from−0\.1325\-0\.1325to−0\.0427\-0\.0427\) but keeps its sign, whereas in GPT\-2\-XL it reverses \(\+0\.1048\+0\.1048\)\. Yet memorised sequences in the pruned GPT\-2\-XL still saturate more per head \(23,56523\{,\}565vs\.17,83617\{,\}836\) despite their higher mean entropy\. Because saturation counts extreme peaks while mean entropy averages over the whole distribution, the pruned model may contain mostly diffuse heads with occasional sharp peaks on memorised content\. We leave testing this hypothesis to future work\.

Finally, the effects are stronger in GPT\-2\-XL, which is pruned considerably more aggressively than Pythia\-2\.8B and shows both larger relative reductions in saturation and the sign reversal in entropy\. Architectural differences, such as positional encoding and the arrangement of attention and MLP blocks, are further candidate explanations, but the present results cannot separate them\.

![Refer to caption](https://arxiv.org/html/2610.40974v1/image.png)\(a\)Before pruning
![Refer to caption](https://arxiv.org/html/2610.40974v1/image1.png)\(b\)After pruning

Figure 1:Average cross\-head agreement across layers among heads exceeding the softmax saturation threshold, before and after pruning\.Figure[1](https://arxiv.org/html/2610.40974#S3.F1)illustrates this coordination effect before and after pruning\. In the baseline model, memorised sequences elicit markedly stronger cross\-head agreement among saturated heads across layers; after pruning, this gap narrows substantially, with memorised and non\-memorised sequences showing more similar coordination profiles\. As with the saturation count above, this comparison is drawn over the pool of heads exceeding the saturation threshold, which itself shrinks after pruning; we have not yet verified how much of the narrowed gap follows from this smaller pool versus a genuine change in coordination among the heads that remain\.

Table 5:Average top\-two logit margin on memorised and non\-memorised sequences before and after head pruning\.The logit margin results in Table[5](https://arxiv.org/html/2610.40974#S3.T5)are consistent with this account: baseline models produce top\-two margins of 8\.79 and 7\.68 on memorised sequences, roughly four times the non\-memorised margins, indicating that the saturation and coordination patterns above co\-occur with anomalously high\-confidence predictions on memorised sequences\. Pruning roughly halves these margins \(to 3\.60 and 3\.86\) while leaving non\-memorised margins nearly unchanged, suggesting a selective effect on the mechanism rather than a general reduction in model confidence\. Aggregated over 20,000 sequences, this per\-step reduction in confidence is plausibly sufficient to cause the model to exit the memorised track frequently, contributing to the large count reductions in Table[1](https://arxiv.org/html/2610.40974#S3.T1)\. We note, as in the Conclusion, that this evidence remains correlational: it establishes that saturation, coordination, and confidence all shift together under pruning, but does not on its own establish that saturation causes the confidence effect, as opposed to both arising from a shared underlying change in the pruned heads’ behaviour\.

## 4Conclusion

We have shown that inference\-time attention head pruning reduces memorised sequence reproduction by 44–94% across three models and two benchmarks, while preserving, and in several cases improving, language modelling performance on the benchmark used to identify redundant heads\. The result holds across pruning objectives that identify different head subsets, indicating that memorisation is distributed across the attention mechanism rather than localised in a narrow set of critical heads\. Mechanistically, memorised sequences induce more frequent and spatially coordinated softmax saturation across heads, translating into anomalously high next\-token confidence that pruning selectively disrupts without degrading language modelling behaviour\. However, the mechanistic account remains correlational; causal intervention analysis, a controlled comparison against non\-memorised sequences matched for predictability, and probing of the MLP stream, which may retain an independent memorisation signal, are left to future work\.

## 5Limitations

This paper’s focus is to demonstrate that inference\-time attention\-head pruning can suppress memorised\-sequence reproduction, and to shed light on the mechanistic role attention saturation plays in that behaviour\. Several limitations should be noted\. Our memorisation evaluation useskk\-extractability under greedy decoding, which captures only one facet of memorisation; a reduced count under this criterion is not a privacy guarantee, since verbatim reproduction is a narrow proxy\([Ippolito et al\., 2023](https://arxiv.org/html/2610.40974#bib.bib18)\)and training data may remain recoverable under sampling\-based decoding, longer prompts, or adversarial extraction\. Sensitivity to decoding strategy, alongside model size, is exactly the kind of robustness question we flag as future work, since establishing it properly requires its own systematic study, and attention\-head pruning itself leaves model weights unchanged, so it should be understood as an inference\-time mitigation rather than unlearning\. The mechanistic account, in turn, draws on theoretical results for attention\-only transformers that do not directly extend to full architectures with residual connections, layer normalisation, and MLP blocks, so our saturation hypothesis is motivated by theory rather than a direct application of it\. Experiments are further restricted to three decoder\-only models of moderate size with documented pretraining corpora, and it is unclear whether the results generalise to larger, instruction\-tuned, or architecturally distinct models; GPT\-Neo’s alternating local and global attention layers, for example, differ from the architectures studied here, and evaluation on such models is left to future work\. Finally, task preservation is assessed on only two benchmarks, WikiText\-103 and LAMBADA, which may not reflect broader capabilities such as factual recall or long\-context reasoning\.

## References

- Bengioet al\.\(2013\)Y\. Bengio, N\. Léonard, and A\. CourvilleEstimating or propagating gradients through stochastic neurons for conditional computation\.arXiv preprint arXiv:1308\.3432\.Cited by:[§2\.3](https://arxiv.org/html/2610.40974#S2.SS3.p1.2)\.
- Bidermanet al\.\(2023a\)S\. Biderman, U\. Prashanth, L\. Sutawika, H\. Schoelkopf, Q\. Anthony, S\. Purohit, and E\. RaffEmergent and predictable memorization in large language models\.Advances in Neural Information Processing Systems36,pp\. 28072–28090\.Cited by:[§2\.1](https://arxiv.org/html/2610.40974#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.40974#S2.SS2.p2.1)\.
- Bidermanet al\.\(2023b\)S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff,et al\.Pythia: a suite for analyzing large language models across training and scaling\.InInternational conference on machine learning,pp\. 2397–2430\.Cited by:[§2\.2](https://arxiv.org/html/2610.40974#S2.SS2.p1.1)\.
- Carliniet al\.\(2022\)N\. Carlini, D\. Ippolito, M\. Jagielski, K\. Lee, F\. Tramer, and C\. ZhangQuantifying memorization across neural language models\.InThe Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.40974#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.40974#S2.SS2.p1.1)\.
- Carliniet al\.\(2021\)N\. Carlini, F\. Tramer, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, U\. Erlingsson,et al\.Extracting training data from large language models\.In30th USENIX security symposium \(USENIX Security 21\),pp\. 2633–2650\.Cited by:[§1](https://arxiv.org/html/2610.40974#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.40974#S2.SS1.p1.1)\.
- Danaet al\.\(2024\)L\. Dana, M\. S\. Pydi, and Y\. ChevaleyreMemorization in attention\-only transformers\.arXiv preprint arXiv:2411\.10115\.Cited by:[§1](https://arxiv.org/html/2610.40974#S1.p2.1)\.
- Gaoet al\.\(2020\)L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima,et al\.The pile: an 800gb dataset of diverse text for language modeling\.arXiv preprint arXiv:2101\.00027\.Cited by:[§2\.2](https://arxiv.org/html/2610.40974#S2.SS2.p2.1)\.
- Ippolitoet al\.\(2023\)D\. Ippolito, F\. Tramer, M\. Nasr, C\. Zhang, M\. Jagielski, K\. Lee, C\. Choquette Choo, and N\. CarliniPreventing generation of verbatim memorization in language models gives a false sense of privacy\.InProceedings of the 16th International Natural Language Generation Conference,C\. M\. Keet, H\. Lee, and S\. Zarrieß \(Eds\.\),Prague, Czechia,pp\. 28–53\.External Links:[Link](https://aclanthology.org/2023.inlg-main.3/),[Document](https://dx.doi.org/10.18653/v1/2023.inlg-main.3)Cited by:[§5](https://arxiv.org/html/2610.40974#S5.p1.1)\.
- Kandpalet al\.\(2022\)N\. Kandpal, E\. Wallace, and C\. RaffelDeduplicating training data mitigates privacy risks in language models\.InInternational Conference on Machine Learning,pp\. 10697–10707\.Cited by:[§1](https://arxiv.org/html/2610.40974#S1.p1.1)\.
- Loshchilov and Hutter \(2017\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[§2\.3](https://arxiv.org/html/2610.40974#S2.SS3.SSS0.Px1.p1.1)\.
- Mahdaviet al\.\(2024\)S\. Mahdavi, R\. Liao, and C\. ThrampoulidisMemorization capacity of multi\-head attention in transformers\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 49942–49967\.Cited by:[§1](https://arxiv.org/html/2610.40974#S1.p2.1),[§2\.4](https://arxiv.org/html/2610.40974#S2.SS4.p2.1)\.
- Menget al\.\(2022\)K\. Meng, D\. Bau, A\. Andonian, and Y\. BelinkovLocating and editing factual associations in gpt\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 17359–17372\.External Links:[Document](https://dx.doi.org/10.52202/068431-1262),[Link](https://proceedings.neurips.cc/paper_files/paper/2022/file/6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.40974#S1.p2.1)\.
- Merityet al\.\(2016\)S\. Merityet al\.The wikitext long term dependency language modeling dataset\.Salesforce Metamind9\.Cited by:[§2\.2](https://arxiv.org/html/2610.40974#S2.SS2.p3.1)\.
- Michelet al\.\(2019\)P\. Michel, O\. Levy, and G\. NeubigAre sixteen heads really better than one?\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2610.40974#S1.p2.1)\.
- Nandaet al\.\(2023\)N\. Nanda, S\. Rajamanoharan, J\. Kramár, and R\. ShahFact finding: attempting to reverse\-engineer factual recall on the neuron level\.External Links:[Link](https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall)Cited by:[§1](https://arxiv.org/html/2610.40974#S1.p2.1)\.
- Papernoet al\.\(2016\)D\. Paperno, G\. Kruszewski, A\. Lazaridou, N\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. FernándezThe lambada dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 1525–1534\.Cited by:[§2\.2](https://arxiv.org/html/2610.40974#S2.SS2.p3.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§2\.2](https://arxiv.org/html/2610.40974#S2.SS2.p1.1),[§2\.2](https://arxiv.org/html/2610.40974#S2.SS2.p2.1)\.
- Siet al\.\(2023\)N\. Si, H\. Zhang, H\. Chang, W\. Zhang, D\. Qu, and W\. ZhangKnowledge unlearning for llms: tasks, methods, and challenges\.arXiv preprint arXiv:2311\.15766\.Cited by:[§1](https://arxiv.org/html/2610.40974#S1.p1.1)\.
- Voitaet al\.\(2019\)E\. Voita, D\. Talbot, F\. Moiseev, R\. Sennrich, and I\. TitovAnalyzing multi\-head self\-attention: specialized heads do the heavy lifting, the rest can be pruned\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 5797–5808\.Cited by:[§1](https://arxiv.org/html/2610.40974#S1.p2.1)\.
- Yuet al\.\(2021\)D\. Yu, S\. Naik, A\. Backurs, S\. Gopi, H\. A\. Inan, G\. Kamath, J\. Kulkarni, Y\. T\. Lee, A\. Manoel, L\. Wutschitz,et al\.Differentially private fine\-tuning of language models\.arXiv preprint arXiv:2110\.06500\.Cited by:[§1](https://arxiv.org/html/2610.40974#S1.p1.1)\.