Fractional State Space Transition for Long Sequence Modeling

Hugging Face Daily Papers Papers

Summary

FRAC是基于分数阶动力学的新型选择性状态空间模型架构,用幂律长记忆替代传统SSM的指数遗忘,通过log间距指数模式和近似实现高效循环模块,并在1.3B参数语言模型上显著提升长上下文性能。

State Space Models (SSMs) compress sequence history into a bounded recurrent state, making the resulting memory law a central architectural choice for long-context performance. Most modern SSMs rely on ODE-based dynamics that lead to exponential forgetting, limiting their ability to retain information over broad temporal ranges. We introduce FRAC, a selective SSM architecture derived from fractional dynamics that replaces this exponential decay with power-law long memory. To make fractional dynamics practical, FRAC approximates the heavy-tailed target kernel with a finite-state, log-spaced sum of exponential modes. This construction turns fractional memory into an efficient recurrent module with parallel training and prefill, while retaining bounded-state autoregressive decoding. Extensive experiments, including 1.3B-parameter language modeling, demonstrate that FRAC consistently improves long-context performance over state-of-the-art SSM baselines while staying competitive on short-context. These results show that fractional dynamics provide a practical and effective prior for long-context SSMs.
Original Article
View Cached Full Text

Cached at: 09/30/26, 08:18 PM

Paper page - Fractional State Space Transition for Long Sequence Modeling

Source: https://huggingface.co/papers/2609.36314

Abstract

StateSpaceModels(SSMs)compresssequencehistoryintoaboundedrecurrentstate,makingtheresultingmemorylawacentralarchitecturalchoiceforlong-contextperformance.MostmodernSSMsrelyonODE-baseddynamicsthatleadtoexponentialforgetting,limitingtheirabilitytoretaininformationoverbroadtemporalranges.WeintroduceFRAC,aselectiveSSMarchitecturederivedfromfractionaldynamicsthatreplacesthisexponentialdecaywithpower-lawlongmemory.Tomakefractionaldynamicspractical,FRACapproximatestheheavy-tailedtargetkernelwithafinite-state,log-spacedsumofexponentialmodes.Thisconstructionturnsfractionalmemoryintoanefficientrecurrentmodulewithparalleltrainingandprefill,whileretainingbounded-stateautoregressivedecoding.Extensiveexperiments,including1.3B-parameterlanguagemodeling,demonstratethatFRACconsistentlyimproveslong-contextperformanceoverstate-of-the-artSSMbaselineswhilestayingcompetitiveonshort-context.Theseresultsshowthatfractionaldynamicsprovideapracticalandeffectivepriorforlong-contextSSMs.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2609\.36314

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.36314 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.36314 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.36314 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models

arXiv cs.LG

本文研究序列模型中的长程记忆资源问题:给定预测记忆律,证明了指数与有限状态模式在预测风险下的匹配上下界,得到达到误差τ所需的 Θ(log²(1/τ)) 状态复杂度定律,并分析了分数长记忆、非线性递归的临界性条件及其与状态空间、门控循环和注意力模型的实验验证。

TIDES: Implicit Time-Awareness in Selective State Space Models

Hugging Face Daily Papers

TIDES 提出一种选择性 SSM 变体,将输入依赖性从步长移到对角状态矩阵上,从而在保持每 token 表达能力的同时原生处理不规则时间戳,并在 UE 分类与 PhysiomeODE 回归基准上取得最佳平均排名。

Time Delay System for Sequence Modeling

arXiv cs.CL

该论文提出一种基于非线性时间延迟系统的序列建模框架,通过可微语义延迟估计器、对角状态空间的闭式 O(1) 解以及可解释的模糊语义调制机制,在文本分类和生成任务上超越了现有最先进方法。

An Input-Dependent Readout Phase for Resonant Sequence Models

arXiv cs.LG

该论文提出让谐振序列模型(ResonatorLM)的读出相位依赖当前 token,在保持 FFT 卷积训练路径的同时引入输入依赖性,使在 256 token 训练、1024 token 测试时困惑度从 12.81 降至 3.76,显著改善长度泛化。