Tag
This theoretical paper studies the non-asymptotic implicit bias of logistic regression under gradient descent, proving that the parameter vector weakly aligns with the max-margin direction quickly, within a doubly exponential number of iterations in the alignment error.
This arXiv paper investigates implicit bias in LLM-based chat AI toward people with intellectual disabilities by generating and analyzing 25,000 stories across five LLMs. Findings reveal negative biases such as infantilization, paternalism, and dependency, highlighting the need for bias mitigation in AI development.
This paper proposes performing PAC-Bayesian analysis on quotient parameter spaces to remove KL contributions from parameter symmetries, and constructs a geometry-induced prior that approximates the ideal posterior-matched prior, resulting in tighter generalization bounds. Experiments on Fourier regression and Query-Key attention show significant reductions in KL divergence and certificate values.
This paper studies the gradient flow dynamics of diagonal linear networks under infinitesimal initialization, generalizing previous results to deep networks and a broader class. It shows that the implicit bias corresponds to a modified ℓ1 norm and identifies the Structural Invariant Manifold as a key geometric structure.
This paper studies the implicit bias of stability-annealed smoothed sign descent for separable linear classification, proving that normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice, and characterizing the rate-indexed barrier path.
MIT researchers show that the edge of stability (EoS) in neural network training is not merely a global optimization phenomenon but selectively redistributes learning across subsets of the training distribution, amplifying progress on some data groups while suppressing others. They identify two key conditions governing this allocation: gradient alignment with the top Hessian eigenvector and sustained non-vanishing gradient magnitude.
This paper studies how depth alone induces an implicit low-rank bias in deep unconstrained feature models trained without regularization, shifting the optimal solution from neural collapse to softmax codes, and provides the first asymptotic and dynamic characterization of this bias under gradient descent with cross-entropy loss.
OpenAI research reveals the 'double descent' phenomenon where test error exhibits a non-monotonic pattern as both model size and training steps increase, challenging traditional understanding of the bias-variance tradeoff in deep learning.