rnn

RWKV is a novel language model architecture that combines the efficiency of RNNs with the parallelizability of Transformers, achieving strong LLM performance with linear time complexity and constant memory footprint.

This paper introduces neural approaches for reversing character-set simplification and abbreviation in medieval text, using one-to-one and banded RNNs trained with self-supervision or parallel corpora, and presents a Python library for letter lemmatization.

FLYNN is a recurrent neural network derived from the fruit fly brain connectome for vision-based robot navigation. It achieves comparable performance to handcrafted networks while showing superior robustness to out-of-distribution data and sensory loss.

A technical analysis comparing memory designs in RNNs, Transformers, and SSMs, arguing that the key question is where to store sequence state rather than which architecture is better. Discusses trade-offs between compressed hidden states, growing KV caches, and synaptic-like memory in model connectivity.

This paper proposes Supervised Memory Training (SMT), which uses Transformer as a super teacher to distill memory states in parallel, then trains RNN with one-step supervised learning, achieving fully parallel training and reducing gradient path from O(T) to O(1), significantly improving long-range dependency learning.

This article introduces a new method proposed by Google Research, Cornell, and USC that takes snapshots of RNN memory and caches them, enabling RNNs to efficiently handle long contexts. It combines Transformer-like strong memory with RNN-like low cost, offering a new direction for long-context AI.

BlinkDL announces RWKV-7 G1g, a pure RNN LLM that claims to be the best in its class and competitive with general LLMs, with high-speed inference on a single RTX 5090.

Created an in-browser WebGPU version of the classic char-rnn demo training on Shakespeare, paying homage to Karpathy joining Anthropic.

Proposes PDRNN, a modular hybrid AI-assisted pedestrian dead reckoning system that combines a recurrent neural network with separate ML models for orientation, velocity, and distance estimation, with optional radio-based stabilization. Experiments on dynamic sports movement data show superior accuracy and precision compared to classic and ML-based methods.

Key-Value Means (KVM) is a novel attention mechanism that combines the strengths of transformers and RNNs with controllable computational complexity and memory usage. It supports fixed-size or growing state, offers subquadratic prefill time and sublinear state growth, and can be implemented without custom kernels.

RL² proposes encoding a fast reinforcement learning algorithm as the weights of a recurrent neural network, learned through slow general-purpose RL, enabling agents to adapt to new tasks with few trials similar to biological learning. The method demonstrates strong performance on both small-scale bandit problems and large-scale vision-based navigation tasks.