Tag
Physics of Language Models Part 4.2 reports 64 from-scratch pretraining runs at 1.3–8B parameters showing that Canon layers save ~30% of pretraining tokens at 8B, that simple gated linear attention with GLA5/GDN2 matches Gated DeltaNet and beats Mamba2, yet standard benchmarks hide failures in basic in-context retrieval and two-hop reasoning even at 8B/1T scale.
Toaster 构建了一种由两个线性记忆级联加激活函数组成的非线性循环记忆,并利用目标传播为隐藏记忆赋值,在 0.2B–1.6B 规模上取得优于 KDA 的损失,且在混合注意力架构中表现最佳。
UniSpark-9B is a unified multimodal model combining full attention with Gated DeltaNet linear attention in a Mixture-of-Transformer-Experts backbone, jointly supporting visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction, achieving 2470.3 on MME-S and 0.83 on GenEval.
Nikita Pavlichenko, former head of pretraining for JetBrains Mellum, systematically evaluated LLM architecture changes since 2020 through 1,129 training experiments. He concluded that architectural gains had largely been exhausted by the DeepSeek-V3 generation, with fine-grained MoE and linear attention representing the most substantial advances, while the impact of training methods such as Muon and inference kernels is often underestimated.
DAMI 提出一种双轴记忆初始化方法,通过将 A_log 按几何级数分布于各头、dt_bias 沿深度呈低-高-低剖面,在 GDN、KDA、GDN2、Mamba-2、RWKV-7 五种线性注意力架构上将 RULER 检索准确率相对随机初始化提升 5.29–16.71,同时不引入额外参数或改变推理开销。
The paper measures dynamic evaluation (taking gradient steps on text being read) across softmax-attention, hybrid linear-attention, and masked-diffusion language models, finding 5–20% next-token loss reductions but showing the adaptation captures style/vocabulary rather than factual retrieval, while low-rank distillation of a model's own conclusions reaches in-context performance without degrading GSM8K.
PACE 是一种免微调的后训练四比特量化器,用于线性注意力循环状态,通过降低量化频率(每隔几个 token 才舍入一次)并采用列对齐的量化尺度来减少误差,从而在 4B-35B 参数混合模型上以约三分之二的内存达到与八比特 DAMP 相当的性能。
The paper builds teams of persistent, discipline-diverse AI researcher agents with private histories and adaptive allocation of experimental budget to autonomously discover linear attention architectures, outperforming GD-NS at 1.3B scale on language modeling and retrieval tasks.
The paper introduces FederatedBytes, a tokenizer-free byte-level small language model with linear attention for efficient multimodal federated learning, outperforming ByT5 and LSTM on sentiment analysis and ScienceQA while reducing memory and communication overheads.
The paper identifies a Competing Learning phenomenon in hybrid language models where recurrence and attention layers compete to learn the same functions, and proposes a staged training procedure that improves language modeling loss, common-sense reasoning, and retrieval tasks on models up to 1B parameters.
介绍基于Kronecker Delta规则的长上下文线性注意力研究(KronLSTM),在长上下文基准上表现优异,并被认为对物理AI与机器人的持续状态记忆具有潜在影响。
这篇论文提出将注意力的指数核展开为 n 项截断 Taylor 级数,并将每项分解为张量幂内积,从而用固定数量的循环状态实现线性时间 Transformer,实测在长上下文下相比 softmax 注意力有显著加速和常数解码延迟。
A researcher trained two budget-matched 15M-parameter 'decision-only' models — a linear recurrent backbone vs a standard Transformer — and found accuracy was a statistical coin flip, while the linear model was 2.37× faster at 8k-token inference and ~6–7× faster to train. The post frames the negative accuracy result as the key finding: small decision models are accuracy-starved regardless of substrate.
A detailed landscape mapping of Post-Transformer architectures for 2026 — including Mamba-2/3, RWKV-7, BDH (Dragon Hatchling), xLSTM, Titans, and various hybrid models — comparing core mechanisms, attention variants, and test-time memory under practical compute and energy budgets.
ReaDy introduces a speculative decoding method that uses hybrid linear attention recurrent states as dynamic context for drafting, achieving a 3.16x decoding speedup over autoregressive generation with Qwen3.8-27B on an NVIDIA B200 while improving throughput and acceptance length over DSpark across seven benchmarks.
论文提出 Kairos——一个基于字节级别的多模态基础模型,结合 LiZAttention2(双向门控 DeltaNet 线性注意力 + 滑动窗口注意力)、稀疏 MoE 前馈层与多尺度字节编解码器,并在极小规模上进行收敛性验证实验,报告编解码器与注意力变体对损失的影响。
The paper introduces NoteBank, a training-free sparse episodic memory that augments linear attention models by archiving selected text and replaying it through native computation, boosting Gated DeltaNet's long-context retrieval accuracy on RULER from 1.7% to 70.3% at one million tokens with negligible FLOPs overhead.
腾讯研究人员在 Hugging Face 上发布了 STEPQuant,一个针对 Delta-rule 循环状态的空间-时间后训练量化框架,用 6-bit 状态即可匹配 FP32 精度,并将服务内存最多削减 68.7%。
V-CoLA 提出一种面向线性注意力混合架构的免训练视觉 token 压缩框架,通过独特性感知的重要性准则与自适应 token 合并策略,在仅保留 50% 视觉 token 时达到 99.5% 原始性能,并实现 1.86x 至 6.15x 的预填充加速。
This paper analyzes the mechanics of long-context hybrid LLMs combining full attention with sliding-window (SWA) or gated linear attention (GLA/GDN), identifying a Seesaw Effect in context extension and proposing Sliding-Window Linear Attention that achieves 16x training-free length extrapolation with 100% NIAH-SK1 accuracy at 64k context.