linear-attention

Tag

Cards List
#linear-attention

Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality

arXiv cs.CL ↗ · yesterday Cached

Physics of Language Models Part 4.2 reports 64 from-scratch pretraining runs at 1.3–8B parameters showing that Canon layers save ~30% of pretraining tokens at 8B, that simple gated linear attention with GLA5/GDN2 matches Gated DeltaNet and beats Mamba2, yet standard benchmarks hide failures in basic in-context retrieval and two-hop reasoning even at 8B/1T scale.

0 favorites 0 likes
#linear-attention

Toaster: Nonlinear Memory from Cascaded Linear Memories and Target Propagation

arXiv cs.LG ↗ · 2d ago Cached

Toaster 构建了一种由两个线性记忆级联加激活函数组成的非线性循环记忆,并利用目标传播为隐藏记忆赋值,在 0.2B–1.6B 规模上取得优于 KDA 的损失,且在混合注意力架构中表现最佳。

0 favorites 0 likes
#linear-attention

UniSpark: Exploring Hybrid Attention for Multimodal Understanding and Generation

Hugging Face Daily Papers ↗ · 2026-11-12 Cached

UniSpark-9B is a unified multimodal model combining full attention with Gated DeltaNet linear attention in a Mixture-of-Transformer-Experts backbone, jointly supporting visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction, achieving 2470.3 on MME-S and 0.83 on GenEval.

0 favorites 0 likes
#linear-attention

@AI_Whisper_X: The gains from LLM model architectures have already been exhausted. I read an article about model architectures: with so many changes to LLM architectures since 2020, which ones actually help when evaluated under the same experimental setup? The author, Nikita Pavlichenko (formerly worked on JetBrains Mellum pretraining…

X AI KOLs Timeline ↗ · 2026-11-02 Cached

Nikita Pavlichenko, former head of pretraining for JetBrains Mellum, systematically evaluated LLM architecture changes since 2020 through 1,129 training experiments. He concluded that architectural gains had largely been exhausted by the DeepSeek-V3 generation, with fine-grained MoE and linear attention representing the most substantial advances, while the impact of training methods such as Muon and inference kernels is often underestimated.

0 favorites 0 likes
#linear-attention

DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention

Hugging Face Daily Papers ↗ · 2026-10-30 Cached

DAMI 提出一种双轴记忆初始化方法,通过将 A_log 按几何级数分布于各头、dt_bias 沿深度呈低-高-低剖面,在 GDN、KDA、GDN2、Mamba-2、RWKV-7 五种线性注意力架构上将 RULER 检索准确率相对随机初始化提升 5.29–16.71,同时不引入额外参数或改变推理开销。

0 favorites 0 likes
#linear-attention

Can Language Models Learn While They Think? Testing Gradient Adaptation Across Architectures

arXiv cs.LG ↗ · 2026-10-28 Cached

The paper measures dynamic evaluation (taking gradient steps on text being read) across softmax-attention, hybrid linear-attention, and masked-diffusion language models, finding 5–20% next-token loss reductions but showing the adaptation captures style/vocabulary rather than factual retrieval, while low-rank distillation of a model's own conclusions reaches in-context performance without degrading GSM8K.

0 favorites 0 likes
#linear-attention

PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States

Hugging Face Daily Papers ↗ · 2026-10-21 Cached

PACE 是一种免微调的后训练四比特量化器,用于线性注意力循环状态,通过降低量化频率(每隔几个 token 才舍入一次)并采用列对齐的量化尺度来减少误差,从而在 4B-35B 参数混合模型上以约三分之二的内存达到与八比特 DAMP 相当的性能。

0 favorites 0 likes
#linear-attention

Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures

Hugging Face Daily Papers ↗ · 2026-10-20 Cached

The paper builds teams of persistent, discipline-diverse AI researcher agents with private histories and adaptive allocation of experimental budget to autonomously discover linear attention architectures, outperforming GD-NS at 1.3B scale on language modeling and retrieval tasks.

0 favorites 0 likes
#linear-attention

FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning

arXiv cs.CL ↗ · 2026-10-19 Cached

The paper introduces FederatedBytes, a tokenizer-free byte-level small language model with linear attention for efficient multimodal federated learning, outperforming ByT5 and LSTM on sentiment analysis and ScienceQA while reducing memory and communication overheads.

0 favorites 0 likes
#linear-attention

Competing Learning in Hybrid Architectures

Hugging Face Daily Papers ↗ · 2026-10-19 Cached

The paper identifies a Competing Learning phenomenon in hybrid language models where recurrence and attention layers compete to learn the same functions, and proposes a staged training procedure that improves language modeling loss, common-sense reasoning, and retrieval tasks on models up to 1B parameters.

0 favorites 0 likes
#linear-attention

Kronecker Delta Rule for Long Context Linear Attention

Hugging Face Daily Papers ↗ · 2026-10-19 Cached

介绍基于Kronecker Delta规则的长上下文线性注意力研究(KronLSTM),在长上下文基准上表现优异,并被认为对物理AI与机器人的持续状态记忆具有潜在影响。

0 favorites 0 likes
#linear-attention

Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States

arXiv cs.LG ↗ · 2026-10-16 Cached

这篇论文提出将注意力的指数核展开为 n 项截断 Taylor 级数,并将每项分解为张量幂内积,从而用固定数量的循环状态实现线性时间 Transformer,实测在长上下文下相比 softmax 注意力有显著加速和常数解码延迟。

0 favorites 0 likes
#linear-attention

I trained a 15M "decision-only" model two ways — linear recurrence vs Transformer attention. Accuracy was a coin flip; latency and training cost weren't.[R]

Reddit r/MachineLearning ↗ · 2026-10-14

A researcher trained two budget-matched 15M-parameter 'decision-only' models — a linear recurrent backbone vs a standard Transformer — and found accuracy was a statistical coin flip, while the linear model was 2.37× faster at 8k-token inference and ~6–7× faster to train. The post frames the negative accuracy result as the key finding: small decision models are accuracy-starved regardless of substrate.

0 favorites 0 likes
#linear-attention

Mapping the Post-Transformer Architecture Landscape in 2026 [D]

Reddit r/MachineLearning ↗ · 2026-10-14

A detailed landscape mapping of Post-Transformer architectures for 2026 — including Mamba-2/3, RWKV-7, BDH (Dragon Hatchling), xLSTM, Titans, and various hybrid models — comparing core mechanisms, attention variants, and test-time memory under practical compute and energy budgets.

0 favorites 0 likes
#linear-attention

ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models

Hugging Face Daily Papers ↗ · 2026-10-14 Cached

ReaDy introduces a speculative decoding method that uses hybrid linear attention recurrent states as dynamic context for drafting, achieving a 3.16x decoding speedup over autoregressive generation with Qwen3.8-27B on an NVIDIA B200 while improving throughput and acceptance length over DSpark across seven benchmarks.

0 favorites 0 likes
#linear-attention

Kairos: Toward a Byte-Level Foundation Model for Every Modality

arXiv cs.LG ↗ · 2026-10-12 Cached

论文提出 Kairos——一个基于字节级别的多模态基础模型,结合 LiZAttention2(双向门控 DeltaNet 线性注意力 + 滑动窗口注意力)、稀疏 MoE 前馈层与多尺度字节编解码器,并在极小规模上进行收敛性验证实验,报告编解码器与注意力变体对损失的影响。

0 favorites 0 likes
#linear-attention

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

Hugging Face Daily Papers ↗ · 2026-10-11 Cached

The paper introduces NoteBank, a training-free sparse episodic memory that augments linear attention models by archiving selected text and replaying it through native computation, boosting Gated DeltaNet's long-context retrieval accuracy on RULER from 1.7% to 70.3% at one million tokens with negligible FLOPs overhead.

0 favorites 0 likes
#linear-attention

Quantization of Linear-Attention (Qwen & Kimi)

Reddit r/LocalLLaMA ↗ · 2026-10-09 Cached

腾讯研究人员在 Hugging Face 上发布了 STEPQuant,一个针对 Delta-rule 循环状态的空间-时间后训练量化框架,用 6-bit 状态即可匹配 FP32 精度,并将服务内存最多削减 68.7%。

0 favorites 0 likes
#linear-attention

V-CoLA: Vision Token Compression with Linear Attention

Hugging Face Daily Papers ↗ · 2026-10-08 Cached

V-CoLA 提出一种面向线性注意力混合架构的免训练视觉 token 压缩框架,通过独特性感知的重要性准则与自适应 token 合并策略,在仅保留 50% 视觉 token 时达到 99.5% 原始性能,并实现 1.86x 至 6.15x 的预填充加速。

0 favorites 0 likes
#linear-attention

Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position

Hugging Face Daily Papers ↗ · 2026-10-07 Cached

This paper analyzes the mechanics of long-context hybrid LLMs combining full attention with sliding-window (SWA) or gated linear attention (GLA/GDN), identifying a Seesaw Effect in context extension and proposing Sliding-Window Linear Attention that achieves 16x training-free length extrapolation with 100% NIAH-SK1 accuracy at 64k context.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback