Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

Hugging Face Daily Papers Papers
linear-rnns attention-mechanism expressivity state-tracking language-modeling open-source

Summary

This paper introduces Complex KDA, an enhanced version of Kimi Delta Attention that combines a delta-rule transformation with a reflection to achieve greater expressivity, outperforming Transformers in some tasks while maintaining efficiency, with open-source code and models available.

Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that composing two delta-rule transitions in a single recurrent update can model a 2D rotation, but this increases the rank and the cost of the updates compared to a single transition. We show that Kimi Delta Attention (KDA) can realize 2D rotations by combining a single delta-rule transformation with a second reflection supplied by its channel-wise gate. This requires extending the parameter ranges of KDA by combining two existing range extensions: allowing gates in [-1,1] and the delta-rule coefficient β in [0,2]. We call the resulting model Complex KDA (CKDA). It preserves KDA's stability and efficiency, with transitions that remain diagonal-plus-rank-one and non-expansive, while reaching the state-tracking expressivity of DeltaProduct_2. We characterize the expressivity of CKDA and prove that every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix. A single CKDA layer can track every finite group isomorphic to a subgroup of SO(3), and many state-tracking results use one fewer layer for CKDA compared to other diagonal-plus-rank-one Linear RNNs. Empirically, combining both extensions yields the strongest length extrapolation among tested KDA range settings on S_3, S_4, and periodic audio continuation. In language modeling, CKDA outperforms Transformers and other linear RNNs, obtains similar results to a KDA baseline, and shows promising scaling behavior. Our code is open source at https://github.com/OpenEuroLLM/ComplexKDA and our models are available at https://huggingface.co/collections/openeurollm/complexkda.
Original Article
View Cached Full Text

Cached at: 09/22/26, 03:29 PM

Paper page - Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

Source: https://huggingface.co/papers/2609.24797 Authors:

,

,

,

,

,

,

,

,

,

Abstract

LinearRNNsbasedonthedelta-ruleenableefficientsequencemodeling,buttheirlinearupdateswithalow-rankcorrectionconstraintheirexpressivity.Priorworkhasshownthatcomposingtwodelta-ruletransitionsinasinglerecurrentupdatecanmodela2Drotation,butthisincreasestherankandthecostoftheupdatescomparedtoasingletransition.WeshowthatKimiDeltaAttention(KDA)canrealize2Drotationsbycombiningasingledelta-ruletransformationwithasecondreflectionsuppliedbyitschannel-wisegate.ThisrequiresextendingtheparameterrangesofKDAbycombiningtwoexistingrangeextensions:allowinggatesin[-1,1]andthedelta-rulecoefficientβin[0,2].WecalltheresultingmodelComplexKDA(CKDA).ItpreservesKDA’sstabilityandefficiency,withtransitionsthatremaindiagonal-plus-rank-oneandnon-expansive,whilereachingthestate-trackingexpressivityofDeltaProduct_2.WecharacterizetheexpressivityofCKDAandprovethateveryorthogonaldiagonal-plus-rank-onematrixisexactlyaCKDAtransitionmatrix.AsingleCKDAlayercantrackeveryfinitegroupisomorphictoasubgroupofSO(3),andmanystate-trackingresultsuseonefewerlayerforCKDAcomparedtootherdiagonal-plus-rank-oneLinearRNNs.Empirically,combiningbothextensionsyieldsthestrongestlengthextrapolationamongtestedKDArangesettingsonS_3,S_4,andperiodicaudiocontinuation.Inlanguagemodeling,CKDAoutperformsTransformersandotherlinearRNNs,obtainssimilarresultstoaKDAbaseline,andshowspromisingscalingbehavior.Ourcodeisopensourceathttps://github.com/OpenEuroLLM/ComplexKDAandourmodelsareavailableathttps://huggingface.co/collections/openeurollm/complexkda.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2609\.24797

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper4

#### openeurollm/complex-kda-1.3B-100B Text Generation• 1B• Updatedabout 5 hours ago • 27 • 1 #### openeurollm/kda-sigmoid-1.3B-100B Text Generation• 1B• Updatedabout 5 hours ago • 8 • 1 #### openeurollm/complex-kda-hybrid-1.3B-100B Text Generation• 1B• Updatedabout 5 hours ago • 8 • 1 #### openeurollm/kda-sigmoid-hybrid-1.3B-100B Text Generation• 1B• Updatedabout 5 hours ago • 13 • 1

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.24797 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.24797 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

You Could Have Come Up with Kimi Delta Attention

Hacker News Top

This blog post derives Kimi Delta Attention step by step from standard softmax attention through linear attention and DeltaNet variants, explaining the state update equations used by recent Qwen and Kimi models.

Kimi-K3 Technical Report [pdf]

Hacker News Top

MoonshotAI releases Kimi-K3, a 2.8T-parameter open-weight multimodal agentic model with a 1M-token context window, built on new Kimi Delta Attention and Attention Residuals architecture, achieving significant scaling improvements.

MoonshotAI/FlashKDA

GitHub Trending (daily)

FlashKDA is a high-performance implementation of Kimi Delta Attention (KDA) kernels built on CUTLASS, optimized for SM90+ GPUs and integrated with flash-linear-attention.