Tag
A scaling study over ~600 training runs finds that full BPTT on looped transformers outperforms truncated alternatives at matched compute, with two fully trained loops emerging as the compute-optimal setting that beats non-looped models with ~27% fewer parameters.
提出 First Passage Attention (FPA)，将注意力遗忘机制从路径求和改为路径极值，包含 Forgetting Transformer 作为特例，并在 110M 语言模型上实现 8 倍训练长度的完美 passkey 检索。
This paper introduces Plausibility-EFG, a group-level disparity metric for measuring explanation plausibility gaps across demographic groups in hate speech detection, and proposes Fair-REF, an adversarial rationale regularization strategy that reduces explanation disparities while maintaining classification performance.
The paper shows that ReLU networks on modular addition can maintain 100% training accuracy and near-flat training cross-entropy while hidden features evolve and held-out accuracy rises above 90%, proving this coexistence with finite-window theorems and reproducing it in deeper networks and transformers.
论文提出 TRACE（Transformer Residual Activation Codebook Encoding），在每个 Transformer 层的后注意力残差处放置向量量化器并预计算码本索引化的键值表示，使分布式边缘设备推理中接收方通过查表直接恢复 K/V，实现通信-计算重叠；在 ViT-Base 与 ViViT-B 上相比最优 VQ 分布式推理框架最高提升 1.89% 精度与 1.27 倍端到端加速，通信开销最高降低 22,624 倍。
TokenGate 是一个轻量级可学习的 token 剪枝模块，通过 token-gated attention 机制在长上下文预填充阶段跳过不重要的 token，实现 1.4–11.3 倍的 TTFT 加速和 1.5–3.4 倍的峰值内存降低且无精度损失。
介绍基于Kronecker Delta规则的长上下文线性注意力研究（KronLSTM），在长上下文基准上表现优异，并被认为对物理AI与机器人的持续状态记忆具有潜在影响。
A retrospective deep dive tracing the evolutionary arc of AI architectures from the single perceptron through early neural nets, the vanishing gradient problem, LSTMs, and the Transformer era, while speculating about the next paradigm shift beyond self-attention.
一篇技术博客，通过"关联记忆"的视角重新推导 Transformer：从线性层的外积记忆，到 MLP 的容量提升，再到自注意力的近乎无界记忆容量，并给出各层可存储记忆数量的启发式估算（如 64 维键可从约 64 条提升到上百万条记忆）。
提出 Selection-Safe Conformal Early Exit 方法，通过终端路径分数与单一 split-conformal 分位数校准，在数据依赖的自适应深度选择下仍保持无分布覆盖保证，无需 α/L 修正，并在 CIFAR-100/ImageNet 等任务上验证了路径选择相比朴素逐层选择的覆盖优势。
论文提出 Optimal Hydra，一种面向 LLM 微调的模型异构联邦学习框架，通过全局层级选择与基于客户端数据分布的注意力头选择构建客户端专属子网络（Hydra-nets），在三个数据集上显著优于基线方法并降低本地资源需求。
这篇论文提出将注意力的指数核展开为 n 项截断 Taylor 级数，并将每项分解为张量幂内积，从而用固定数量的循环状态实现线性时间 Transformer，实测在长上下文下相比 softmax 注意力有显著加速和常数解码延迟。
提出了一种新的可扩展性原则——'有效连贯认知速度'(S_C^eff)，用于衡量AI系统在现实变化速度超出训练分布时维持可靠行为的能力，并展示了受意识机制启发的Co⁴架构相比Transformer和gated-MLP基线在此指标上表现更优。
The paper proposes Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head alongside the standard LM head to let models explicitly point to and copy tokens from context, mitigating the LM head gradient bottleneck. Pretrained models from 135M to 1.7B parameters show better training efficiency, downstream performance, and long-context operating points with negligible overhead.
This paper investigates whether syntactically specialized attention heads (subject-verb heads) causally drive Transformer language models' syntactic predictions, finding mixed to negative evidence that these heads play a causal role despite their specialized attention patterns.
Titan-Llama 提出一种后训练方法，通过分层记忆架构和测试时参数更新，将预训练注意力模型改造为具备外部神经记忆模块的长上下文高效推理模型，在 QuALITY 和 LongHealth 上显著提升准确率并降低推理延迟与显存占用。
该研究探讨多智能体LLM系统中应如何通信以聚合分布式私有信息，证明在保持校准不确定性的情况下，通信概率信念比通信具体动作更能促进协作决策，并提出了一种事后对话校准干预方法。
提出 First Passage Attention（FPA），用路径极值替代单调衰减累积，从而在压缩 KV 缓存的同时实现完美的长程检索；在 110M 语言模型上以约 25% 缓存达到 8 倍训练长度的 passkey 检索。
A researcher trained two budget-matched 15M-parameter 'decision-only' models — a linear recurrent backbone vs a standard Transformer — and found accuracy was a statistical coin flip, while the linear model was 2.37× faster at 8k-token inference and ~6–7× faster to train. The post frames the negative accuracy result as the key finding: small decision models are accuracy-starved regardless of substrate.
提出了 Continuum Attention，一种用连续分段线性记忆曲线上的积分替代离散键值查找的因果 Transformer 注意力算子，在语言建模困惑度上优于标准 softmax 与 focal attention。