regularization

This paper proposes QQWorld, a quantile-quantile matching objective that replaces the Epps-Pulley objective in LeWorldModel for better regularization of latent distributions, improving planning success in control environments.

Proposes Unbiased Open World Regularization (UOWReg), an encoder-only framework that enforces conditional distribution matching to achieve statistical independence between learned representations and sensitive attributes, reducing bias while maintaining accuracy.

Explains SIGReg, a novel regularizer for JEPA that prevents representation collapse by forcing embeddings to follow an isotropic Gaussian distribution, with theoretical guarantees and a clean training loop.

This paper introduces a regularization technique for Direct Alignment Algorithms (DAAs) that maintains normalized response probabilities, mitigating over-optimization and likelihood displacement. The method improves generation quality and benchmark performance, achieving over 20% relative increase on AlpacaEval2 and 9% gains on general benchmarks for Llama-3.1-8B-Instruct.

This paper introduces OrthoGrad, a geometric intervention that removes the radial component of weight gradients during optimization, and shows that it reduces memorization of noisy labels in small-data regimes but does not prevent eventual memorization.

This paper identifies that standard ridge regularization in potential recovery from flow on directed graphs collapses and reverses the ordering of the estimate due to gauge dependence. It proposes a gauge-invariant Dirichlet energy penalty that yields a parameter-insensitive solution and demonstrates robust dynamic range preservation on real clickstream data, with implications for preventing oversmoothing in graph neural networks.

Google Research shows that the creativity of diffusion models is a mathematical consequence of neural network regularization causing score smoothing and interpolation, demystifying their ability to generate novel data rather than just memorize.

This paper introduces Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that uses evaluative feedback to improve alignment in imitation learning, achieving up to 98% reduction in misalignment in Safety Gymnasium environments.

This paper proposes FedFMX, a Fisher-Routed Mixture of Experts framework for Federated Class-Incremental Learning, addressing capacity conflict, catastrophic forgetting, and data heterogeneity via adaptive expert specialization.

Derives the closed-form gradient of the Wolkowicz-Styan upper bound on the loss Hessian eigenspectrum to guide neural network training toward flat minima, and introduces Hessian Spectral Range (HSR) Regularization. Numerical experiments show that HSR narrows the Hessian eigenvalue range, avoids sharp minima and saddle points, and achieves flat solutions comparable to Sharpness-Aware Minimization (SAM).

Introduces VISReg, a regularization method for JEPA (Joint Embedding Predictive Architecture) training that combines variance, invariance, and sketching constraints.

A Twitter thread listing 13 core machine learning concepts that interviewers expect candidates to know, covering topics from bias-variance tradeoff to the curse of dimensionality.

This paper introduces LISA, a regularization method that aligns the intermediate features of a side network with an approximated likelihood score to improve training efficiency and the quality of visual-condition controllable generation in score-based generative models.

EMAgnet introduces parameter-space exponential moving average regularization for policy gradient self-play in large two-player zero-sum games, achieving lower exploitability compared to uniform regularization targets.

This paper presents a large-scale empirical study of the Derivative Regularization (DREG) penalty, showing it achieves high accuracy and noise robustness, particularly with GELU activation and data-scarce regimes, positioning it as a general-purpose plug-and-play regularizer for neural networks.

A research paper that combines a small amount of human demonstrations as a regularization objective with self-play reinforcement learning, enabling human-compatible driving policies using far less human data (30 minutes vs thousands of hours) and training in 15 hours on a single consumer GPU.

This paper introduces Tiered Language Models (TLMs), which allow a single set of open-weight model parameters to support multiple capability levels controlled by secret keys. The method enables selective exposure of private capabilities while preserving public model behavior and resisting extraction.

The authors found that regularizing ColBERT models fixes the efficient ANN methods MUVERA and SMVE, which had broken on modern ColBERT models, simplifying multi-vector retrieval infrastructure in an unexpected way.

This paper provides guidance on the appropriate use of different Schatten-p norms in deep learning, analyzing their theoretical properties and practical implications for model regularization and optimization.

This paper studies data-constrained language model pretraining, proposing masked-input regularization (MIR) to improve validation loss and downstream performance, and SoftQ, a scaling law that better captures model-data interaction under repeated data.