Metraipolis › archive › dc_4bcf3ebb5929

The Extender: A Log-Structured Transformer — arXiv:2609.32759

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32759
License (as archived)
CC0-1.0
Archived
2026-10-07T21:00:36.552Z
SHA-256 of the archived text
4208ded37303f0c8f57317d908a93a0b7f8db58a13da00c5b23859f06ecaa3c0
Stable URL of this page
https://metraipolis.com/documento/dc_4bcf3ebb5929

How to cite

The Extender: A Log-Structured Transformer — arXiv:2609.32759 — arXiv (en), https://arxiv.org/abs/2609.32759 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4bcf3ebb5929 (Metraipolis archive dc_4bcf3ebb5929, sha256 4208ded37303f0c8f57317d908a93a0b7f8db58a13da00c5b23859f06ecaa3c0)
BibTeX
@misc{dc4bcf3ebb5929,
  title = {The Extender: A Log-Structured Transformer — arXiv:2609.32759},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32759},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4bcf3ebb5929; sha256 4208ded37303f0c8f57317d908a93a0b7f8db58a13da00c5b23859f06ecaa3c0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32759. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We introduce the Extender, a log-structured variant of the standard Transformer architecture. In a standard Transformer, each layer communicates with subsequent layers exclusively via the residual $\mathbf{h}$, a superposition channel. The Extender adds a concatenation channel $\mathbf{x}$: each layer $\ell$ emits both a residual update $\delta_\ell$ which is added to $\mathbf{h}$, and a much smaller extension $\epsilon_\ell$ which is appended to $\mathbf{x}$. While both the FFN and $\mathbf{q}$ see $\mathbf{h}$, the attention $\mathbf{kv}$ projections take only $\mathbf{x}$ as input. As a result, the fully extended $\mathbf{x}$ contains the complete input for the $\mathbf{kv}$ projections of all layers, reducing the persistent attention memory footprint from $2Ld_{model}$ to $\sum|\epsilon_\ell|$. We find that with $|\epsilon_\ell|=32$, the Extender matches Transformer accuracy on short-context (CORE) tasks at 199M-924M parameters, and exceeds Transformer accuracy on long-context (RULER) workloads, again at 924M parameters. For our 1664-wide, 924M model, the Extender's persistent attention memory footprint is $104\times$ smaller than MHA. The memory savings grow with model width.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.32759 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.453Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).