The Extender: A Log-Structured Transformer

Hugging Face Daily Papers Papers
transformer architecture attention memory-efficiency long-context log-structured

Summary

提出了 Extender，一种日志结构化的 Transformer 架构，通过增加一条拼接通道 ε 而非仅依赖残差通道 h，在 924M 参数规模下实现与 Transformer 相当的短上下文精度、更优的长上下文（RULER）表现，且持久化注意力内存足迹较多头注意力缩小 104 倍。

We introduce the Extender, a log-structured variant of the standard Transformer architecture. In a standard Transformer, each layer communicates with subsequent layers exclusively via the residual h, a superposition channel. The Extender adds a concatenation channel x: each layer ell emits both a residual update δ_ell which is added to h, and a much smaller extension ε_ell which is appended to x. While both the FFN and q see h, the attention kv projections take only x as input. As a result, the fully extended x contains the complete input for the kv projections of all layers, reducing the persistent attention memory footprint from 2Ld_{model} to sum|ε_ell|. We find that with |ε_ell|=32, the Extender matches Transformer accuracy on short-context (CORE) tasks at 199M-924M parameters, and exceeds Transformer accuracy on long-context (RULER) workloads, again at 924M parameters. For our 1664-wide, 924M model, the Extender's persistent attention memory footprint is 104times smaller than MHA. The memory savings grow with model width.
Original Article
View Cached Full Text

Cached at: 10/05/26, 08:50 PM

Paper page - The Extender: A Log-Structured Transformer

Source: https://huggingface.co/papers/2609.32759

Abstract

WeintroducetheExtender,alog-structuredvariantofthestandardTransformerarchitecture.InastandardTransformer,eachlayercommunicateswithsubsequentlayersexclusivelyviatheresidualh,asuperpositionchannel.TheExtenderaddsaconcatenationchannelx:eachlayerellemitsbotharesidualupdateδ_ellwhichisaddedtoh,andamuchsmallerextensionε_ellwhichisappendedtox.WhileboththeFFNandqseeh,theattentionkvprojectionstakeonlyxasinput.Asaresult,thefullyextendedxcontainsthecompleteinputforthekvprojectionsofalllayers,reducingthepersistentattentionmemoryfootprintfrom2Ld_{model}tosum|ε_ell|.Wefindthatwith|ε_ell|=32,theExtendermatchesTransformeraccuracyonshort-context(CORE)tasksat199M-924Mparameters,andexceedsTransformeraccuracyonlong-context(RULER)workloads,againat924Mparameters.Forour1664-wide,924Mmodel,theExtender’spersistentattentionmemoryfootprintis104timessmallerthanMHA.Thememorysavingsgrowwithmodelwidth.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.32759

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.32759 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.32759 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.32759 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Recurrent Looped Transformer

Hugging Face Daily Papers

论文提出 Recurrent Looped Transformer (RLT)，将层分为并行因果编码器和循环解码器，使计算路径随序列长度增长而每 token 成本固定，在奇偶校验、S5 置换追踪和模运算等算法任务上显著优于标准 Transformer 的长度外推能力。

Causal Latent Workspaces for Looped Transformers

arXiv cs.CL

该论文提出因果潜在工作区 Transformer（causal latent workspace Transformer），通过在共享深度的循环块之间加入可读写的前缀专属潜在槽位，提升了循环 Transformer 的语言建模困惑度，并在三个字节级语料上的多数设置中优于五个基线对照。

Your Transformer Is Secretly Elastic

Hugging Face Daily Papers

提出一种免重训练的方法，将预训练 Transformer 转化为可弹性伸缩的模型，通过联合排序并软剪枝单个 MLP 神经元和注意力头，在多种稀疏度目标下连续自适应计算预算，在 27 个视觉与文本 Transformer 编码器上实现最高 60% 稀疏度下的平滑性能衰减。

You could have invented transformers from memory

Hacker News Top

一篇技术博客，通过"关联记忆"的视角重新推导 Transformer：从线性层的外积记忆，到 MLP 的容量提升，再到自注意力的近乎无界记忆容量，并给出各层可存储记忆数量的启发式估算（如 64 维键可从约 64 条提升到上百万条记忆）。