Metraipolis › archive › dc_2d2eb2907eba

Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention — arXiv:2609.24797

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.24797
License (as archived): CC0-1.0
Archived: 2026-10-09T07:07:56.207Z
SHA-256 of the archived text: 8c1d2e6978266f4a5b7acfbdc21e9e3bc7e7f74ceb19493bac914ed13121cb98
Stable URL of this page: https://metraipolis.com/documento/dc_2d2eb2907eba

How to cite

Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention — arXiv:2609.24797 — arXiv (en), https://arxiv.org/abs/2609.24797 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2d2eb2907eba (Metraipolis archive dc_2d2eb2907eba, sha256 8c1d2e6978266f4a5b7acfbdc21e9e3bc7e7f74ceb19493bac914ed13121cb98)

BibTeX

@misc{dc2d2eb2907eba,
  title = {Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention — arXiv:2609.24797},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.24797},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2d2eb2907eba; sha256 8c1d2e6978266f4a5b7acfbdc21e9e3bc7e7f74ceb19493bac914ed13121cb98},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24797. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that composing two delta-rule transitions in a single recurrent update can model a 2D rotation, but this increases the rank and the cost of the updates compared to a single transition. We show that Kimi Delta Attention (KDA) can realize 2D rotations by combining a single delta-rule transformation with a second reflection supplied by its channel-wise gate. This requires extending the parameter ranges of KDA by combining two existing range extensions: allowing gates in $[-1,1]$ and the delta-rule coefficient $\beta$ in $[0,2]$. We call the resulting model Complex KDA (CKDA). It preserves KDA's stability and efficiency, with transitions that remain diagonal-plus-rank-one and non-expansive, while reaching the state-tracking expressivity of DeltaProduct$_2$. We characterize the expressivity of CKDA and prove that every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix. A single CKDA layer can track every finite group isomorphic to a subgroup of $\mathrm{SO}(3)$, and many state-tracking results use one fewer layer for CKDA compared to other diagonal-plus-rank-one Linear RNNs. Empirically, combining both extensions yields the strongest length extrapolation among tested KDA range settings on $S_3$, $S_4$, and periodic audio continuation. In language modeling, CKDA outperforms Transformers and other linear RNNs, obtains similar results to a KDA baseline, and shows promising scaling behavior. Our code is open source at this https URL and our models are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.24797 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:58:09.449Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).