Tag
The article compares the routes of two AI model architecture camps, Mamba and RWKV: Mamba targets cloud-side enterprise AI platforms, while RWKV targets on-device and edge devices, each adapting to different hardware and inference scenarios.
RWKV is a novel language model architecture that combines the efficiency of RNNs with the parallelizability of Transformers, achieving strong LLM performance with linear time complexity and constant memory footprint.
BlinkDL announces RWKV-7 G1g, a pure RNN LLM that claims to be the best in its class and competitive with general LLMs, with high-speed inference on a single RTX 5090.
Ali Hatamizadeh announces Gated DeltaNet-2, a new linear attention model that outperforms KDA and Mamba-3 at 1.3B scale; @BlinkDL_AI notes its recurrence is nearly identical to RWKV-7's DPLR.