Tag
该论文研究高维场景下的持续线性回归，推导了梯度下降有限步数训练时泛化损失的闭式表达，并证明最优步长按 lnT/T 衰减，其效果渐近等价于最优 L2 正则化。研究还探讨了对角正则化条件以及任务变异性与隐式正则化之间的权衡，并通过实验验证了理论预测。
该论文证明，仅靠在发布时移除信息内容（以互信息衡量）无法普遍保证开源权重模型的抗篡改能力——函数保持的重参数化会改变梯度下降几何结构，使得即使互信息为零，攻击者也可能在一次梯度步骤内恢复被移除的能力。作者提出认证需要对攻击动力学施加约束，而非仅依赖发布时的互信息。
This paper analyzes four growth strategies for constructive classifiers (soft decision trees and related models) under a unified protocol, proving that the natural way to deepen a soft decision tree leaves new gates with identically zero gradients, making them unable to learn — and showing a small random perturbation fixes it.
A tweet promoting a 100% open-source tool that applies gradient descent-style optimization to AI agents' memory.
A new paper introduces Matryoshka Attribution, a method using gradient descent to identify responsible parts of neural networks, achieving top performance on the Mechanistic Interpretability Benchmark.
This paper introduces the first method for continuous gradient descent optimization in machine learning models with p-adic parameters, using the Berkovich affine line to enable effective learning on tasks like modular arithmetic.
This article explains why the backpropagation algorithm in neural networks is implemented in a backward pass, clarifying the efficiency advantages over a forward computation based on the chain rule.
This paper introduces Priority-Constrained Descent (PCD), a gradient-based optimization framework for hierarchical multi-objective problems that preserves primary descent while allowing minimal distortion for secondary objectives.
A new dynamic quantization method called Voodoo Quant, now open-sourced under MIT license, uses gradient descent to optimize per-tensor quant levels for AI models, showing better performance at aggressive quant levels.
The author presents a method to create anamorphic sculptures by optimizing 3D meshes using gradient descent and differentiable rendering, demonstrated with a pig silhouette formed from car, pig, and pigeon meshes.
The paper proposes REAL-Q, a novel end-to-end aligned post-training quantization method for large language models that uses dynamic gradient descent to mitigate quantization errors, achieving significant improvements in KL divergence reduction over state-of-the-art methods.
This paper re-examines the edge of stability in deep learning optimization, proposing a new formulation based on directional Hessian and gradient-alignment score for more accurate predictions and diagnostic tools.
The paper explores whether any neural network can be redesigned to train effectively with gradient descent, proving a universality result that for any network, there exists an extension that reproduces given weights and outputs via gradient descent.
This paper establishes convergence guarantees for gradient descent on general feedforward neural networks of arbitrary width/depth, using a novel generalized Lipschitz smoothness condition that holds for common activations and mean-squared error, without special initialization or dataset requirements.
This paper proposes Adaptive Hybrid PSO (AHPSO), which uses a sigmoid function on swarm diversity to automatically modulate gradient influence during search. Results show it outperforms standard PSO and rivals CMA-ES on certain problem classes, but the advantage is not universal.
A theory paper introducing Decoupled Descent (DD), a training method that uses approximate message passing Onsager corrections to enforce asymptotic equality between training and test error during gradient descent, potentially enabling better stopping and hyperparameter tuning.
A theoretical study shows that in overparameterized ReLU networks, a positive-homogeneous scaling gauge hidden in the initial parameters can deterministically control which duplicate neuron learns a teacher feature, affecting specialization time and pruning trajectories.
This theoretical paper studies the non-asymptotic implicit bias of logistic regression under gradient descent, proving that the parameter vector weakly aligns with the max-margin direction quickly, within a doubly exponential number of iterations in the alignment error.
This paper investigates why Adam does not exhibit gradient descent's implicit low-rank bias in factored models, showing that coordinate-wise preconditioning breaks the relevant symmetry, while shared-scalar methods like Muon and Shampoo preserve it.
This paper proposes a regularity-aware stochastic multi-gradient descent method (MoRe) that adaptively switches between conflict-avoidant and scalarization updates. The method achieves improved convergence rates from O~T^{-1/4} to O~T^{-1/2} in nonconvex settings while maintaining per-iterate conflict avoidance.