linear-attention

Tag

Cards List
#linear-attention

Agens Volundr 32B Preview: our small team's first model on our own hybrid architecture. Only 18 of 72 layers keep a KV cache (Apache-2.0)

Reddit r/LocalLLaMA ↗ · 2026-10-05

Blockway released Agens Volundr 32B Preview, an Apache-2.0 32B dense model on a hybrid architecture where only 18 of 72 layers keep a KV cache, aiming for efficient long-context inference on consumer/local GPUs. The team reports competitive benchmarks versus Qwen3.8-27B on coding and math but weaker agentic performance, with a custom sglang build required for now.

0 favorites 0 likes
#linear-attention

HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models

Hugging Face Daily Papers ↗ · 2026-10-05 Cached

HLA-WM提出一种训练免费的混合线性注意力框架,通过几何引导的检索与递归线性状态重组合来缓解GatedDeltaNet在长时视频世界模型中的远距离遗忘问题,在60秒基准上以12倍内存节省同时提升所有一致性与相机控制指标。

0 favorites 0 likes
#linear-attention

Retrieval-Centric Deep Learning in Growing Nonparametric Neural Networks

Hugging Face Daily Papers ↗ · 2026-10-02 Cached

本文提出“检索中心深度学习”(RCDL)范式,通过为每个训练样本存储键值对并在推理时用注意力机制检索重组,构建不断增长的非参数神经网络,并为基于RBF与softmax核的注意力层推导了基于函数梯度的学习规则。实验展示了其在图像分类和合成师生任务上的潜力,并揭示了MesaNet/DeltaNet等线性注意力变体与近期优化器的形式化联系。

0 favorites 0 likes
#linear-attention

LOCI: Spatial Linear Memory for Streaming World Models

Hugging Face Daily Papers ↗ · 2026-09-30 Cached

LOCI introduces a hybrid spatial memory architecture for streaming video world models, combining key-value caches with a recurrent linear-attention memory conditioned on camera geometry. It reproduces revisited content more faithfully than full-softmax baselines while reducing peak memory by about 30% and enabling constant-memory streaming of long videos.

0 favorites 0 likes
#linear-attention

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Hugging Face Daily Papers ↗ · 2026-09-29

The paper proposes STEPQuant, a spatial-temporal post-training quantization framework for delta-rule recurrent states in linear attention models, achieving near-FP32 accuracy at 6-bit precision, over 5x state compression, and up to 68.7% serving memory reduction when integrated into SGLang.

0 favorites 0 likes
#linear-attention

Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

Hugging Face Daily Papers ↗ · 2026-09-29 Cached

The paper introduces triadic linear attention, which generalizes linear attention by writing a triadic outer product of two keys and a value into a third-order tensor state, enabling an E-fold state-size increase with only two extra projections. Applied to Gated DeltaNet and scalar-gated linear attention, it improves long-context language modeling and recall, outperforming alternative state-enlargement methods.

0 favorites 0 likes
#linear-attention

GTR: Gated Token Recurrence for Efficient Dense Prediction

Hugging Face Daily Papers ↗ · 2026-09-23 Cached

The paper introduces GTR, a softmax-free recurrent vision backbone using gated linear attention that achieves 58.9 box AP on COCO with ~1.9 ms latency, distilled from a DINOv3 detection teacher and efficient for high-resolution dense prediction and edge deployment.

0 favorites 0 likes
#linear-attention

Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation

Hugging Face Daily Papers ↗ · 2026-09-17 Cached

Video DeltaNet presents a hybrid attention mechanism combining Softmax and linear attention to enhance efficiency in video generation models, achieving a 14.5x speedup over baseline methods.

0 favorites 0 likes
#linear-attention

Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks It

arXiv cs.LG ↗ · 2026-09-16 Cached

This paper decomposes associative recall in fixed-state recurrences, finding convolution and curriculum learning key to performance, and proposes interventions to address interference rather than capacity limitations.

0 favorites 0 likes
#linear-attention

SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization

Hugging Face Daily Papers ↗ · 2026-09-13 Cached

SpectralShift introduces a spectral reparameterization approach to effectively extend the context window of Gated DeltaNet models by reshaping the decay spectrum, improving long-context capabilities through continual pretraining.

0 favorites 0 likes
#linear-attention

Kalman Delta Networks: Uncertainty-aware Associative Memory

Hugging Face Daily Papers ↗ · 2026-09-07 Cached

Introduces Kalman Delta Networks, which improve language modeling by reformulating linear attention as a linear-Gaussian state-space model with Kalman-filter updates to track memory uncertainty, yielding efficient approximations that outperform existing linear-attention models.

0 favorites 0 likes
#linear-attention

Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design

arXiv cs.LG ↗ · 2026-09-04 Cached

This paper proposes intervention-based metrics to differentiate retrieval and positional heads in RoPE Transformers, leading to a principled hybrid architecture (HwH) that combines full and linear attention for improved language modeling and long-context extrapolation.

0 favorites 0 likes
#linear-attention

Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

Hugging Face Daily Papers ↗ · 2026-09-03 Cached

This paper explains why fully quantizing the recurrent Gated DeltaNet layers in hybrid LLMs to 4-bit NVFP4 preserves accuracy across long-context benchmarks, detailing mechanisms like outlier localization and robust delta-rule dynamics.

0 favorites 0 likes
#linear-attention

Test Time Training (3 minute read)

TLDR AI ↗ · 2026-09-03 Cached

The article discusses Test Time Training as a potential new scaling axis in AI development, analyzing a paper that frames it as a form of linear attention and exploring its implications for model training and continual learning.

0 favorites 0 likes
#linear-attention

Sliding-window beats linear attention

Hugging Face Daily Papers ↗ · 2026-08-28 Cached

Sliding window attention with sinks outperforms post-trained linear attention on long-context tasks without retraining, offering a cheaper and more reliable inference solution for LLMs.

0 favorites 0 likes
#linear-attention

@davsca1: Check out our #ECCV2026 paper "Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention", where …

X AI KOLs Timeline ↗ · 2026-08-26 Cached

This paper introduces Spatially-Sparse Linear Attention for low-latency object detection with event cameras, achieving state-of-the-art accuracy with 20x less computation than prior asynchronous methods.

0 favorites 0 likes
#linear-attention

Qwen 4 architecture: What do we know?

Reddit r/LocalLLaMA ↗ · 2026-08-25

The author speculates on the possible architecture of Qwen 4, discussing potential designs like embedding-offloaded linear attention or multi-head latent attention hybrids based on reverse engineering Deepseek.

0 favorites 0 likes
#linear-attention

The Query Knows What to Forget: A Second Erase Direction for Linear Attention

arXiv cs.LG ↗ · 2026-08-17 Cached

Introduces a Query-derived Erase Direction (QED) for linear attention models to improve retrieval and about double the usable context length, as demonstrated on the S-NIAH-1 benchmark.

0 favorites 0 likes
#linear-attention

Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

Hugging Face Daily Papers ↗ · 2026-08-12 Cached

This paper presents the first systematic study of massive activations in hybrid linear-attention LLMs, uncovering pre-attention spikes and inter-spike plateaus governed by cancellation timing, and showing how their morphology recovers at full-attention limits.

0 favorites 0 likes
#linear-attention

Retrofitting Linear Attention into Diffusion Language Models

arXiv cs.LG ↗ · 2026-08-10 Cached

This paper introduces block-hybrid attention, which combines exact softmax attention within active denoising blocks and linear attention over previous blocks, to accelerate inference in pretrained diffusion language models. The authors retrofit this hybrid attention into LLaDA 2.1, achieving up to 1.7x higher decoding throughput with minimal post-training.

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback