Tag
The paper proposes REAL-Q, a novel end-to-end aligned post-training quantization method for large language models that uses dynamic gradient descent to mitigate quantization errors, achieving significant improvements in KL divergence reduction over state-of-the-art methods.
This paper re-examines the edge of stability in deep learning optimization, proposing a new formulation based on directional Hessian and gradient-alignment score for more accurate predictions and diagnostic tools.
The paper explores whether any neural network can be redesigned to train effectively with gradient descent, proving a universality result that for any network, there exists an extension that reproduces given weights and outputs via gradient descent.
This paper establishes convergence guarantees for gradient descent on general feedforward neural networks of arbitrary width/depth, using a novel generalized Lipschitz smoothness condition that holds for common activations and mean-squared error, without special initialization or dataset requirements.
This paper proposes Adaptive Hybrid PSO (AHPSO), which uses a sigmoid function on swarm diversity to automatically modulate gradient influence during search. Results show it outperforms standard PSO and rivals CMA-ES on certain problem classes, but the advantage is not universal.
A theory paper introducing Decoupled Descent (DD), a training method that uses approximate message passing Onsager corrections to enforce asymptotic equality between training and test error during gradient descent, potentially enabling better stopping and hyperparameter tuning.
A theoretical study shows that in overparameterized ReLU networks, a positive-homogeneous scaling gauge hidden in the initial parameters can deterministically control which duplicate neuron learns a teacher feature, affecting specialization time and pruning trajectories.
This theoretical paper studies the non-asymptotic implicit bias of logistic regression under gradient descent, proving that the parameter vector weakly aligns with the max-margin direction quickly, within a doubly exponential number of iterations in the alignment error.
This paper investigates why Adam does not exhibit gradient descent's implicit low-rank bias in factored models, showing that coordinate-wise preconditioning breaks the relevant symmetry, while shared-scalar methods like Muon and Shampoo preserve it.
This paper proposes a regularity-aware stochastic multi-gradient descent method (MoRe) that adaptively switches between conflict-avoidant and scalarization updates. The method achieves improved convergence rates from O~T^{-1/4} to O~T^{-1/2} in nonconvex settings while maintaining per-iterate conflict avoidance.
Introduces SymExpLin (SEL), a weight reparameterization that combines symmetric-exponential and linear pathways to improve optimization in neural networks, reducing training steps by up to 1.49x on transformers.
This paper provides worst-case convergence analyses for Schedule-Free gradient descent and stochastic gradient descent in nonconvex optimization, establishing optimal rates and strict-saddle avoidance, thus theoretically justifying their empirical success.
This tweet lists 7 fundamental math ideas used daily by ML engineers, with brief explanations emphasizing the underlying derivations, such as why gradient descent moves in the steepest direction and why softmax plus cross-entropy yields a clean gradient.
This paper demonstrates that optimal learning rate scaling in deep scalar linear networks is inherently data-dependent, contradicting prior data-agnostic scaling rules. It shows that with data-dependent scaling, convergence becomes depth-independent, including at infinite depth.
Proposes a hybrid least squares/gradient descent method for MIONets to accelerate training by using alternating least squares for the last layer parameters of multiple branch networks, leveraging Kronecker and Khatri-Rao products.
A tweet promoting a curated collection of math and deep learning resources for understanding the foundations behind models like Claude, including linear algebra, real analysis, optimization, and representation theory.
A blog post discussing optimization techniques for constrained categorical probability distributions, using softmax reparameterization and log barrier methods, applied to protein binder design.
A project that visualizes gradient descent for different optimization algorithms, useful for understanding how optimizers work in machine learning.
FastMix is a novel framework that automates data mixture discovery for training large models using a single proxy model and bilevel optimization, achieving state-of-the-art performance with significant efficiency gains.
This paper analyzes generalization error, uniform stability, and uniform argument stability of gradient descent (GD) and stochastic gradient descent (SGD) over discrete parameter spaces with deterministic or stochastic rounding, showing that rounding degrades generalization for GD and introduces dimension-dependent errors for stochastic rounding.