面向多维序列建模的 Tucker Bottleneck Attention

arXiv cs.LG 论文
attention-mechanism tensor-decomposition efficient-transformers video-prediction weather-forecasting arxiv

摘要

提出 Tucker Bottleneck Attention（TuBA），利用低秩张量结构实现多维序列的高效全局 token 混合，在视频预测和全球天气预报上以显著更低的计算成本取得更优精度，最高减少 85.1% 计算量并实现 4.27 倍加速。

arXiv:2610.09090v1 Announce Type: new Abstract: The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.
查看原文
查看缓存全文

缓存时间: 2026/10/08 12:53 

# Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling
Source: [https://arxiv.org/abs/2610.09090](https://arxiv.org/abs/2610.09090)
[View PDF](https://arxiv.org/pdf/2610.09090)[HTML \(experimental\)](https://arxiv.org/html/2610.09090v1)

> Abstract:The quadratic cost of self\-attention limits scalability to long sequences from multidimensional data\. We introduce Tucker bottleneck attention \(TuBA\), which exploits low\-rank tensor structure for efficient global token mixing\. TuBA projects hidden tensors into compact Tucker cores, performs multi\-head self\-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation\. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores\. On video prediction and global weather forecasting, TuBA achieves favorable accuracy\-efficiency trade\-offs over standard and efficient attention and task\-specific models\. Compared to standard self\-attention, TuBA reduces error and computation by up to 24\.7% and 66\.6% for video prediction and 37\.1% and 85\.1% for autoregressive weather forecasting, with speedups up to 4\.27 times\. Low\-rank Tucker cores and multi\-frame generation also outperform full\-rank attention and frame\-by\-frame generation, respectively\.

## Submission history

From: Ryan Solgi \[[view email](https://arxiv.org/show-email/5e428576/2610.09090)\] **\[v1\]**Tue, 6 Oct 2026 20:45:21 UTC \(835 KB\)

相似文章

HLA-WM：面向长时视频世界模型的混合线性注意力

Hugging Face Daily Papers

HLA-WM 提出一种免训练的混合线性注意力框架，通过几何引导的检索与递归线性状态重组，缓解 GatedDeltaNet 在长时视频世界模型中的远距离遗忘问题。在 60 秒基准上，该方法以 12 倍内存节省同时提升了所有一致性与相机控制指标。