Metraipolis › archive › dc_752fc26d8d94
Zero-Gated Cross-Attention — arXiv:2611.01116
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.01116
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:36:43.291Z
- SHA-256 of the archived text
88d4020daf11591b900014d8797dd8ab991eeacb3d5848ee31dc972650ce3c6a- Stable URL of this page
https://metraipolis.com/documento/dc_752fc26d8d94
How to cite
Zero-Gated Cross-Attention — arXiv:2611.01116 — arXiv (en), https://arxiv.org/abs/2611.01116 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_752fc26d8d94 (Metraipolis archive dc_752fc26d8d94, sha256 88d4020daf11591b900014d8797dd8ab991eeacb3d5848ee31dc972650ce3c6a)
BibTeX
@misc{dc752fc26d8d94,
title = {Zero-Gated Cross-Attention — arXiv:2611.01116},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.01116},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_752fc26d8d94; sha256 88d4020daf11591b900014d8797dd8ab991eeacb3d5848ee31dc972650ce3c6a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01116. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Injecting non-text modalities into frozen language models often introduces early optimization instability, making it difficult to separate architectural gains from transient training dynamics. We introduce Inductively Gated Cross-Attention (IGCA), a zero-initialized cross-attention bridge that preserves pretrained model behavior at initialization while enabling gradual multimodal conditioning. Integrated across six decoder layers of frozen Falcon-40B, open_llama_30b, and Bloom-7b1 backbones (with 40,404,961,288 trainable adapter parameters used during training), IGCA acts as an exact algebraic no-op at initialization with a maximum output discrepancy of 0.00e+00. Over 59,500 training steps on Falcon-40B, the training loss decreases from 3.9403 to 0.0007 (with final ten step losses of [0.0001, 0.0003, 0.0005, 0.0002, 0.0005, 0.0011, 0.0005, 0.0006, 0.0005, 0.0007]). On held-out evaluation, the best gate-on checkpoint achieves a loss of 2.2201 compared to 4.3469 for the gate-off baseline -- a 48.9% reduction (2.1268 nats) -- while average held-out loss improves from 4.3423 to 2.2498 (1.9075 nats) with a mean gate magnitude of $\vert{}\tanh(\alpha)\vert{} = 0.0713$.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.01116 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:12:54.478Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).