ssm

SHiPPO extends HiPPO by transporting polynomial projection coefficients into a moving channel frame, enabling selective state-space models to recover order-sensitive memory signals. The paper provides theoretical foundations and diagnostics supporting its transported-memory prior.

A technical analysis comparing memory designs in RNNs, Transformers, and SSMs, arguing that the key question is where to store sequence state rather than which architecture is better. Discusses trade-offs between compressed hidden states, growing KV caches, and synaptic-like memory in model connectivity.

The article introduces Raven, a new State Space Model (SSM) with selective memory allocation that achieves state-of-the-art performance on recall tasks and demonstrates superior length generalization compared to existing models like SWA.