self-attention

This paper proposes a timing-aware, interpretable self-attention framework that encodes wearable behavioral signals as discrete events with onset timestamps and durations, outperforming existing event-based approaches while improving interpretability for task transition detection.

《Mathematical Theory of the Transformer》是一部长达825页的数学专著，系统阐述了Transformer架构的理论基础，涵盖嵌入、自注意力、矩阵分解、谱分析、信息几何、训练动力学与优化等主题。

一篇技术博客，通过"关联记忆"的视角重新推导 Transformer：从线性层的外积记忆，到 MLP 的容量提升，再到自注意力的近乎无界记忆容量，并给出各层可存储记忆数量的启发式估算（如 64 维键可从约 64 条提升到上百万条记忆）。

该论文提出将 Transformer 网络解释为混合概率推理过程的可学习近似：自注意力对应在输入条件化、动态结构的马尔可夫随机场上执行 max-product 信念传播，而层间深度演进则对应隐式的期望最大化（EM）过程。基于此视角，作者提出 meta-model 范式，将自注意力、混合专家和门控单元等机制统一起来。

该论文研究语言模型自注意力机制的实际检索能力，通过仅保留每层每头注意力权重最高的令牌，估计在给定损失容忍度下保持性能所需的有效注意力集大小，发现较小集即可接近全注意力基线，且注意力集大小随上下文增长而增加，即使没有更多信息需要检索。

An interactive diagram comparing self-attention and cross-attention mechanisms in AI models, published as part of an educational library on attention.

This paper proposes Jacobian-guided noise injection, a training strategy that improves quantization robustness in large language models by injecting noise into pre-attention logits with variance derived from the Jacobian norm, leading to significant performance gains in low-bit quantization settings.

This paper investigates why cross-direction pairing in bidirectional LSTMs underperforms same-direction pairing for dependency relation-type classification, using frozen-trunk diagnostics to analyze representational redundancy and directional information decay.

This study proposes a ring-based Spatial Transformer to model non-linear spatial interactions between building distribution and pedestrian flow around railway stations in Tokyo, outperforming Geographically Weighted Regression and challenging compact city assumptions in urban planning.

This paper analyzes how the parameterization of query-key and output-value circuits in self-attention models affects attention sharpness during training. Through gradient-flow analysis, they show that faster query-key learning relative to output-value learning produces sharper attention, improving interpretability without sacrificing predictive performance.

This paper proposes a novel framework for causal discovery in multivariate time series using inverted self-attention mechanisms, introducing a causal self-attention module (CSAM) and a global causal algorithm to identify latent causal links and reduce spurious correlations.

HOMIE is a framework for human-object centric video personalization that integrates MLLM features to improve subject fidelity and interaction patterns, achieving state-of-the-art performance.

EMAGN is a new efficient multi-attention graph network for traffic forecasting that linearizes self-attention via learned clustering, reducing complexity from quadratic to linear while maintaining accuracy close to full-attention models, with significant reductions in training time, inference time, and GPU memory.

This paper proposes PTTSD, a probabilistic framework for depression severity detection from clinical interview transcripts that models uncertainty and provides temporal interpretability, achieving competitive performance on benchmark datasets.

Promotes an educational resource explaining Transformer architecture, covering token embeddings, self-attention, residual connections, and connections to GPT and BERT.

This paper by Brandon Sandhu provides a mathematically rigorous yet accessible derivation of the Transformer architecture, covering tokenization, embeddings, attention mechanisms, and other core components, with prerequisites in linear algebra, calculus, probability, and information theory.

This paper analyzes hallucination in large language models as a structural consequence of three architectural decisions: self-attention's co-occurrence learning, maximum likelihood estimation training objective, and autoregressive decoding's left-to-right commitment. It maps each mechanism to specific hallucination types and argues that dataset pathologies amplify but do not cause these vulnerabilities.

Introduces Kuramoto attention, a self-attention layer where hidden states are phase angles on a torus, enabling synchronization through gated cosine similarity and circular mean updates. The layer performs comparably to standard transformers on character-level language modeling.

Introduces Contribution Weights, a projection-based metric that accounts for attention weight, value magnitude, and directional alignment to more faithfully measure token importance in transformer LLMs, revealing active functional roles of attention sinks.

This paper presents a mathematical framework for Transformer dynamics as a nonlinear control system on probability measures, proving that Gaussian distributions remain Gaussian under the flow, reducing to finite-dimensional bilinear control, and establishing reachability conditions and asymptotic stability results.