标签
TileMix引入了一种以分块为中心的混合精度注意力机制，以加速大型语言模型中的长上下文预填充，通过将分数分块组路由到FP16或INT8路径来平衡准确性和效率。
本文提出Q-Interference，一种用于GPT模型的内存高效量子启发注意力机制，该机制使用相位感知评分和精确三角因式分解，以改善令牌交互而不增加内存开销。
一项可重复性研究揭示了在Pre-LN Transformer中移除残差连接时产生的非对称效应：移除注意力跳跃导致崩溃，而移除FFN跳跃在较小规模下允许部分恢复。
ARGUS是一个被动式Wi-Fi感知系统，它利用注意力引导的Transformer从信道状态信息中识别人员，无需附加设备，在降低计算成本的同时实现了高精度。
本文介绍了Wiola，一个具有13M参数的仅解码器语言模型，采用螺旋旋转位置编码和门控螺旋注意力等新组件，以提升小型设备端语言模型的参数效率。
本文提出了注意力感知变换编码（AATC）用于压缩大语言模型中的KV缓存，通过注意力机制最小化失真，在约5.8倍压缩下实现了近乎无损的准确率。
SSOG-Attention 引入了一种新的注意力机制，使用可分离高斯和来实现亚二次复杂性，在性能上优于或匹配SDPA，并具有更快的收敛速度和更高的效率。
本文探讨语言模型中的注意力机制如何无需选择性门控即可使潜在变量变得可访问，并识别出一个特定需求的中等深度窗口，注意力介导的聚集在此发生。
This paper proves that a single normalized nonnegative kernel-attention head requires exponentially many features to solve a simple Min-IP task on three-token sequences, whereas dense softmax attention solves it with constant temperature and m-dimensional scores, highlighting a fundamental expressive-power gap between kernel and full attention.
本文提出了混合门控注意力（HyGA），一个包含三种门控策略的框架，能够提升注意力的表示能力并缓解注意力汇聚（attention sinks）问题，在性能和效率上均优于标准门控注意力。
This paper introduces Power Law Graph Attention (PLGA) and the PLDR-LLM architecture, an exact generalization of scaled dot-product attention using input-generated bilinear operators. It presents theoretical results including an inference-collapse theorem, empirical stability measurements, and machine-checked proofs in Lean 4.
PhysAttNet 是一种物理信息注意力框架，通过领域引导的正则化增强轻量级 CNN 预测器，以提高工业与天体物理时间序列预测的准确性和泛化能力。
AoH is a data-free method that identifies retrieval and streaming heads from the spectral geometry of query-key projections, enabling sparse attention without runtime attention scores. At 50% sparsity it retains 96.5% of full-attention performance while reducing prefill/decode latency and KV-cache memory.
TaskSense 提出了一种以任务为中心的世界建模框架，利用基于先前潜在状态的条件随机空间注意力以及辅助逆动力学目标，聚焦于与控制相关的区域，相比 DreamerV3，在视觉干扰下具有更强的鲁棒性。
This paper introduces Matrix Zonotopic Attention (MZAttn), a context-adaptive value projection for set transformers, and provides a theoretical analysis showing it can represent target operators that standard attention requires depth to approximate. Experiments demonstrate performance gains on high-rank combinatorial set-prediction tasks.
The paper introduces DUD (Decoupled Update Dynamics), a framework that separates Feed-Forward Network and Attention contributions via causal interventions to improve uncertainty quantification and calibration in large language models, outperforming state-of-the-art baselines.
This paper applies graph signal processing to analyze how LLMs internally represent numerical sequences during in-context learning, finding that attention-induced token graphs and hidden-state signals show systematic, context-dependent signatures related to input complexity.
ATFlash introduces a per-RoPE-wavelength distance window that prunes query-key inner-product terms proportional to each frequency pair's wavelength, cutting 37-48% of attention compute with minimal quality loss and up to 1.31x speedups on long-context LLM inference.
AttnLink is a research paper presenting an attention-based framework for schema linking in Text-to-SQL, converting LLM internal attention into continuous relevance scores for schema items. Experiments on Spider, BIRD, and Spider2-SQLite show high mAP scores and millisecond-scale latency.
本文识别了ALiBi位置编码中的一种数值下溢故障模式，该模式可能使注意力权重归零，分析其影响，并评估了使用ALiBi训练模型的缓解策略。