@_albertgu: Introducing a new sequence model Raven which pushes the boundary of fixed-state-size sequence models! Raven bridges pop…
Summary
Researchers introduce Raven, a novel sequence model that merges state space model efficiency with a selective slot-updating mechanism inspired by sliding window attention to improve long-context retrieval. The approach offers a more principled alternative to existing linear-time models.
Similar Articles
@rshia_afz: 1/ SSMs struggle on recall benchmarks due to their fixed-size state. But are current models actually storing context “w…
The article introduces Raven, a new State Space Model (SSM) with selective memory allocation that achieves state-of-the-art performance on recall tasks and demonstrates superior length generalization compared to existing models like SWA.
RAVEN: A Regime-Aware Variable-context Expert Network for Financial Time Series Forecasting
The paper proposes RAVEN, a Mixture-of-Experts framework that adaptively determines temporal context windows for each input sample to handle non-stationary financial time series. It achieves state-of-the-art performance on financial and traffic benchmarks.
RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO
RAVEN introduces a real-time autoregressive video extrapolation framework with CM-GRPO, a novel reinforcement learning method for consistency model sampling, improving long-horizon generation quality.
The KV-cache wall: why fixed-size memory sequence models keep coming back
Explores the growing memory bottleneck of KV-cache in transformer inference, explaining why alternative architectures with fixed-size memory like Mamba and RWKV are gaining renewed attention.
Impressive! Openclaw solved complex raven like problems.
Openclaw, an open-source AI model, reportedly outperforms ChatGPT on complex Raven's Progressive Matrices problems, though it refuses to reveal its internal chain-of-thought.