Tag
本文提出“检索中心深度学习”（RCDL）范式，通过为每个训练样本存储键值对并在推理时用注意力机制检索重组，构建不断增长的非参数神经网络，并为基于RBF与softmax核的注意力层推导了基于函数梯度的学习规则。实验展示了其在图像分类和合成师生任务上的潜力，并揭示了MesaNet/DeltaNet等线性注意力变体与近期优化器的形式化联系。
This paper quantifies the memorization-to-generalization transition (grokking) in neural networks through scaling laws, revealing that data complexity is the primary driver of transition time compared to model capacity.
This paper presents an exponential depth hierarchy for ReLU networks in terms of L2 approximation error, demonstrating that deeper networks offer exponentially improved representational power for function approximation.
This paper explores optimization landscapes in complex neural networks using Kähler geometry and information manifolds, providing theoretical guarantees on descent paths and analyzing effects of Calabi-Yau metrics.
A theoretical study shows that in overparameterized ReLU networks, a positive-homogeneous scaling gauge hidden in the initial parameters can deterministically control which duplicate neuron learns a teacher feature, affecting specialization time and pruning trajectories.
This paper studies the gradient flow dynamics of diagonal linear networks under infinitesimal initialization, generalizing previous results to deep networks and a broader class. It shows that the implicit bias corresponds to a modified ℓ1 norm and identifies the Structural Invariant Manifold as a key geometric structure.
MIT researchers show that the edge of stability (EoS) in neural network training is not merely a global optimization phenomenon but selectively redistributes learning across subsets of the training distribution, amplifying progress on some data groups while suppressing others. They identify two key conditions governing this allocation: gradient alignment with the top Hessian eigenvector and sustained non-vanishing gradient magnitude.
This paper theoretically demonstrates that two-layer neural networks trained on group composition tasks learn spectral representations, with neurons converging to irreducible representations and achieving rotational rank-one alignment, providing a representation-theoretic account of feature learning.
This paper provides a theoretical analysis of how neural networks learn structured representations during group composition tasks, proving that training dynamics drive neurons to converge to irreducible group representations with exponential convergence rates. The work establishes a representation-theoretic account of feature learning and characterizes a low-rank compression phenomenon for matrix-valued group representations.
This paper identifies a collapse-and-refine mechanism in diffusion models under the manifold hypothesis, proposing Score-induced Latent Diffusion (SiLD) that provably avoids the curse of dimensionality. Experiments show SiLD matches or outperforms VAE-based latent diffusion models.
This paper argues that robust state tracking in recurrent models depends on error control dynamics rather than just expressive capacity, proving that affine recurrent networks suffer from accumulating errors that limit their effective horizon.
The article explores the Universal Approximation Theorem in deep learning, analyzing the representation capacity of individual neurons and neural network layers using ReLU activation functions.