Metraipolis › archive › dc_5da351d77f55
Wiola 13M, a Gated Spiral Attention Architecture for Parameter Efficient Small Language Models — arXiv:2608.14604
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.14604
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T17:47:48.284Z
- SHA-256 of the archived text
8765d5556632a43d3436094df78ef260eb106c9269a0b3b9b133283c26cacef3- Stable URL of this page
https://metraipolis.com/documento/dc_5da351d77f55
How to cite
Wiola 13M, a Gated Spiral Attention Architecture for Parameter Efficient Small Language Models — arXiv:2608.14604 — arXiv (en), https://arxiv.org/abs/2608.14604 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5da351d77f55 (Metraipolis archive dc_5da351d77f55, sha256 8765d5556632a43d3436094df78ef260eb106c9269a0b3b9b133283c26cacef3)
BibTeX
@misc{dc5da351d77f55,
title = {Wiola 13M, a Gated Spiral Attention Architecture for Parameter Efficient Small Language Models — arXiv:2608.14604},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.14604},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5da351d77f55; sha256 8765d5556632a43d3436094df78ef260eb106c9269a0b3b9b133283c26cacef3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14604. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Small language models in the ten to one hundred million parameter range are attractive for on device inference, rapid experimentation, and controlled scientific study, yet most of them reuse the standard transformer block without adaptation to the small scale regime. We present Wiola, a decoder only language model whose novelty is concentrated in three drop in components of every layer. First, Spiral Rotary Positional Encoding perturbs the standard rotary frequencies by a slowly growing per dimension factor so that phase trajectories fan outward, improving long range discrimination while adding no parameters. Second, Gated Spiral Attention introduces a per head, content adaptive scalar gate derived from a causal cumulative statistic of the query stream, providing an implicit and differentiable form of soft head selection at negligible cost. Third, the Butterfly feed forward block replaces the conventional expansion layer with a multiplicative interaction and an intra block bypass path, matching the parameter count of a four times gated linear unit block while improving gradient flow in shallow stacks. We formalize each component, derive exact parameter and computation budgets, and prove that the gated attention admits an exact and numerically verified equivalence between full sequence training and cached autoregressive decoding, so that no approximation is introduced at inference time. We also describe a fully reproducible training and evaluation protocol on a standard tiny story corpus. The reference implementation is released as an open source package with weights ready publishing support. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.14604 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T16:29:13.907Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).