Tag
《Mathematical Theory of the Transformer》是一部长达825页的数学专著，系统阐述了Transformer架构的理论基础，涵盖嵌入、自注意力、矩阵分解、谱分析、信息几何、训练动力学与优化等主题。
This paper introduces Anchor Divergences, a method that leverages the interplay between contrastive learning, exponential families, and information geometry to define context-specific semantic geometries on fixed CLIP-like representations, allowing similarity to reflect task-relevant attributes (e.g., dog breed vs. background) instead of a single fixed cosine geometry. Retrieval experiments show the approach is both effective and efficient for specifying context-dependent semantic similarity.
This paper analyzes Adam as a diagonal empirical Fisher approximation to natural gradient descent, measuring its geometric deviation across linear, logistic, and small neural network loss landscapes. Results show deviation grows under ill-conditioning but Adam still reaches low loss, suggesting its effectiveness comes from approximation-error balance and momentum smoothing rather than true NGD tracking.
The paper introduces an information-efficiency ratio (IER) for optimizing token selection in sparse on-policy distillation, demonstrating that using only 1% of tokens can achieve performance comparable to full supervision.
This paper introduces a new parameter pruning scheme for neural networks based on Fisher information distances, which outperforms existing methods on MNIST and CIFAR-10 datasets across fully-connected networks and vision transformers.
This paper uses Fisher–Rao geometry to show that large language models share behavioral structures that are learned and controllable, enabling transfer learning and minimal-disturbance interventions.
This paper explores optimization landscapes in complex neural networks using Kähler geometry and information manifolds, providing theoretical guarantees on descent paths and analyzing effects of Calabi-Yau metrics.
A second-order early warning signal for multi-turn prompt injection is introduced, based on information geometry and a statistical manifold. The method uses a meta rate derived from the second derivative of the stability parameter to predict adversarial trajectories before threshold crossing, providing proactive detection.
Introduces CuBAS, an information-geometric framework for adaptive data selection in supervised classification that uses local curvature of the data manifold to identify informative samples, achieving improved accuracy across 30 benchmark datasets.
This paper introduces the degeneracy distillery, a method that automatically detects and resolves degenerate parameter combinations in physical models by estimating and flattening the Fisher information matrix, reducing the simulation budget required for neural posterior estimation while providing physical insight.
CoreMem proposes a resource-efficient edge-cloud memory architecture for dialogue agents, using Riemannian retrieval with a Fisher-Rao metric and Fisher-guided discrete token distillation to achieve strong accuracy improvements within an 8 GB VRAM budget.
Introduces Fisher width, a Riemannian analogue of Gaussian width for statistical manifolds, which captures local statistical curvature and is invariant under reparameterization. The paper develops its theory, proves generalization bounds for Fisher-Lipschitz classes, and demonstrates computable estimators on MNIST.
This paper studies generalization in learning through the lens of bounded-rational decision theory, where the learner's response law induces a tradeoff between training loss and sample dependence. The authors show that this tradeoff is governed by an f-divergence regularizer and that generalization can be certified from the learner's hedging behavior.