delta-rule

腾讯研究人员在 Hugging Face 上发布了 STEPQuant，一个针对 Delta-rule 循环状态的空间-时间后训练量化框架，用 6-bit 状态即可匹配 FP32 精度，并将服务内存最多削减 68.7%。

The paper proposes STEPQuant, a spatial-temporal post-training quantization framework for delta-rule recurrent states in linear attention models, achieving near-FP32 accuracy at 6-bit precision, over 5x state compression, and up to 68.7% serving memory reduction when integrated into SGLang.

This paper introduces Complex KDA, an enhancement to Kimi Delta Attention that improves the expressivity of linear RNNs by enabling 2D rotations and state-tracking capabilities, achieving superior length extrapolation and language modeling performance.

The paper proposes DeltaMomentum, a key-value based momentum update rule that adapts forgetting rates based on input frequency, demonstrating faster convergence in neural network training across various scales.

Proposes Erase-then-Delta Attention (EDA), a memory update rule for linear attention that decouples erase and write addresses to selectively suppress stale information before writing new content. Experiments on 2.5B dense and 25B MoE models demonstrate consistent gains in standard and long-context evaluations.

Gated DeltaNet-2 introduces separate erase and write gates for linear attention, achieving superior performance in long-context language modeling and retrieval tasks.

Proposes delta-Mem, a lightweight online memory mechanism that uses a compact state matrix updated by delta-rule learning to improve long-context performance of frozen LLMs without full fine-tuning or context extension.

The paper introduces δ-mem, a lightweight online memory mechanism that augments frozen LLMs with a compact associative memory state updated by delta-rule learning, achieving significant improvements on memory-heavy benchmarks without fine-tuning or context extension.