rwkv

RWKV is a novel language model architecture that combines the efficiency of RNNs with the parallelizability of Transformers, achieving strong LLM performance with linear time complexity and constant memory footprint.

BlinkDL announces RWKV-7 G1g, a pure RNN LLM that claims to be the best in its class and competitive with general LLMs, with high-speed inference on a single RTX 5090.

Ali Hatamizadeh announces Gated DeltaNet-2, a new linear attention model that outperforms KDA and Mamba-3 at 1.3B scale; @BlinkDL_AI notes its recurrence is nearly identical to RWKV-7's DPLR.