Tag
This paper shows that zeroing out 729 task-redundant attention heads in GPT-2-XL at inference time reduces verbatim memorisation by 91.5% while slightly improving WikiText-103 perplexity, offering a training-free way to mitigate privacy leakage from LLMs.
This paper analyzes integer-sequence benchmarks for language models using minimum description length, revealing that these benchmarks often measure memorisation rather than inductive reasoning, and introduces a new difficulty measure.