memorization

This paper introduces OrthoGrad, a geometric intervention that removes the radial component of weight gradients during optimization, and shows that it reduces memorization of noisy labels in small-data regimes but does not prevent eventual memorization.

This paper introduces analytically tractable Bayesian information restricted diffusion (BIRD) models to study the memorization-generalization phase transition in diffusion models, finding that generation proceeds near the edge of memorization and that information restriction helps circumvent the curse of dimensionality.

This paper investigates the extent to which language models memorize training data, exploring implications for privacy and security.

This paper proposes Native Factorized Weights for transformers, where every linear layer is trained as a product of two low-rank matrices from initialization. Experiments show a corpus-determined optimal rank that minimizes validation loss and a generalization band, outperforming dense baselines with fewer parameters.

This paper investigates parametric memorization in tabular foundation models that use in-context learning, introducing a probing framework (IclMem) to separate context-based predictions from memorization. It finds moderate memorization signals under specific conditions but notes they largely vanish under realistic training scenarios.

This paper uses statistical mechanics to explain the relationship between machine learning and memorization.

This paper systematically studies the damage caused by exact document repetition during language model pretraining, showing that repeating a moderately sized subset a moderate number of times maximally harms performance, and that repetition can waste up to 33% of compute (as measured by compute-equivalent loss).

Cyclic denoising is introduced as a novel extraction attack that reveals ultrastable memorized training images in diffusion models by repeatedly noising and denoising samples. The technique requires no gradients or weight inspection and has implications for privacy auditing.

Presents output vector editing, a constrained-optimization weight edit to mitigate memorization in LLMs by modifying MLP neuron output vectors instead of zeroing activations, achieving up to 87.9% suppression with minimal locality failures.

This paper introduces DiRL, a direction-aware reinforcement learning framework that distinguishes reasoning-driven diversity from memorization-driven diversity in LLM exploration. It extracts an internal reasoning-memorization direction from model representations and shapes rewards to prioritize reasoning-aligned exploration, showing improvements on math and general reasoning benchmarks.

This paper introduces the concept of deployment-time memorization in foundation-model agents, analyzing how memory design choices (summarization aggressiveness, retrieval breadth, deletion mode) affect personalization utility, extraction risk, and deletion fidelity, proposing new metrics such as Personalization Recall, Adversarial Extraction Rate, and Forgetting Residue Score.

PropMe is a propensity-aware framework for evaluating LLM memorization, distinguishing between forced reproduction capabilities and natural propensity using SimpleTrace for deterministic attribution across open models and datasets.

The paper proposes a method to mitigate spurious correlations by disentangling learning dynamics of core and spurious features using a two-stage sample scoring function, achieving state-of-the-art debiasing performance with only 10% of training data.

This paper investigates memorization in diffusion models and finds that they preferentially memorize prototypical examples with common substrings, even after deduplication, and that early stopping leads to an overproduction of common motifs, dubbed 'slop'.

This paper introduces NumLeak, a framework for detecting when foundation models memorize public numeric benchmarks from pretraining rather than demonstrating out-of-sample skill, and shows that top LLMs recall values like Fama-French returns with high fidelity, proposing a simple system-prompt defense.

This paper introduces infilling extraction, a new method for extracting training data from diffusion language models by using arbitrary binary masks, showing that such models are more vulnerable to memorization attacks than previously thought.

This paper introduces a novel task, transitive inference with exceptions, and analytically characterizes how neural network models (kernel ridge regression) balance relational generalization and memorization. The theory is validated in pretrained language models, showing systematic mistakes predicted by the theory.

This paper studies how fill-in-the-middle (FIM) pretraining affects verbatim memorization, finding that FIM more often recovers short spans while standard left-to-right training recovers long exact continuations, and that memorization under FIM grows linearly with repetitions.

This paper introduces Zero-CoT Probe (ZCP), a black-box detection method that identifies evasive data contamination in LLMs by truncating chain-of-thought reasoning and comparing performance on perturbed datasets, achieving robust detection of both direct and indirect contamination.

Vocabi is a tool that helps users translate, save, and memorize words while they read.