Tag
This paper proposes Supervised Memory Training (SMT), which uses Transformer as a super teacher to distill memory states in parallel, then trains RNN with one-step supervised learning, achieving fully parallel training and reducing gradient path from O(T) to O(1), significantly improving long-range dependency learning.
Introduces PACI, a bubble-free asynchronous pipeline parallel training method that bounds forward/backward weight inconsistency using local gradient accumulation, achieving higher throughput and faster time-to-accuracy without sacrificing stability or memory usage.
This paper investigates parallel-in-time algorithms for training recurrent neural networks in dynamical systems reconstruction, proposing GTF-DEER that enables stable learning over long sequences and improves reconstruction accuracy.
This paper introduces the Structured Recurrent Mixer (SRM), an architecture enabling algebraic conversion between parallel training and recurrent inference without specialized kernels. Experiments show SRMs achieve significantly higher throughput and concurrency compared to Transformers, with effective performance in reinforcement learning tasks.