rnn

本研究在受控因子实验（720次运行）中比较了门控初始化（常规 vs. chrono）与循环dropout/zoneout在LSTM和GRU上学习长程依赖的效果，发现chrono初始化在合成任务上显著优于常规初始化，而dropout通常延迟学习、zoneout仅在部分条件下有益。

该论文评估了基于循环神经网络(RNN)的Sentinel-2时间序列树种分类方法，并使用积分梯度(Integrated Gradients)进行可解释性分析，重点研究了纯林与混交林挪威云杉像元的区分，最佳RNN模型Macro-F1达0.851，优于XGBoost和随机森林。

A Show HN project showcasing an RNN-based system that generates Korean Hangul handwriting strokes, celebrating Hangul Day with an open-source creative deep learning demo.

A comprehensive visual guide to language models for text classification, including hands-on experiments with RNNs, CNNs, transformers, and calibration techniques.

The paper introduces triadic linear attention, which generalizes linear attention by writing a triadic outer product of two keys and a value into a third-order tensor state, enabling an E-fold state-size increase with only two extra projections. Applied to Gated DeltaNet and scalar-gated linear attention, it improves long-context language modeling and recall, outperforming alternative state-enlargement methods.

The article shares an intuition that compaction in RNNs represents agentic recurrence while recursion in RLMs represents agentic attention, comparing different context-handling approaches in AI models.

RWKV is a novel language model architecture that combines the efficiency of RNNs with the parallelizability of Transformers, achieving strong LLM performance with linear time complexity and constant memory footprint.

This paper introduces neural approaches for reversing character-set simplification and abbreviation in medieval text, using one-to-one and banded RNNs trained with self-supervision or parallel corpora, and presents a Python library for letter lemmatization.

FLYNN is a recurrent neural network derived from the fruit fly brain connectome for vision-based robot navigation. It achieves comparable performance to handcrafted networks while showing superior robustness to out-of-distribution data and sensory loss.

A technical analysis comparing memory designs in RNNs, Transformers, and SSMs, arguing that the key question is where to store sequence state rather than which architecture is better. Discusses trade-offs between compressed hidden states, growing KV caches, and synaptic-like memory in model connectivity.

This paper proposes Supervised Memory Training (SMT), which uses Transformer as a super teacher to distill memory states in parallel, then trains RNN with one-step supervised learning, achieving fully parallel training and reducing gradient path from O(T) to O(1), significantly improving long-range dependency learning.

This article introduces a new method proposed by Google Research, Cornell, and USC that takes snapshots of RNN memory and caches them, enabling RNNs to efficiently handle long contexts. It combines Transformer-like strong memory with RNN-like low cost, offering a new direction for long-context AI.

BlinkDL announces RWKV-7 G1g, a pure RNN LLM that claims to be the best in its class and competitive with general LLMs, with high-speed inference on a single RTX 5090.

Created an in-browser WebGPU version of the classic char-rnn demo training on Shakespeare, paying homage to Karpathy joining Anthropic.

Proposes PDRNN, a modular hybrid AI-assisted pedestrian dead reckoning system that combines a recurrent neural network with separate ML models for orientation, velocity, and distance estimation, with optional radio-based stabilization. Experiments on dynamic sports movement data show superior accuracy and precision compared to classic and ML-based methods.

Key-Value Means (KVM) is a novel attention mechanism that combines the strengths of transformers and RNNs with controllable computational complexity and memory usage. It supports fixed-size or growing state, offers subquadratic prefill time and sublinear state growth, and can be implemented without custom kernels.

RL² proposes encoding a fast reinforcement learning algorithm as the weights of a recurrent neural network, learned through slow general-purpose RL, enabling agents to adapt to new tasks with few trials similar to biological learning. The method demonstrates strong performance on both small-scale bandit problems and large-scale vision-based navigation tasks.