Metraipolis › archive › dc_8be2fa3509c8

Local Prototype Reconstruction for Text-Compatible Speech-to-LLM Bridge Pretraining — arXiv:2610.11159

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.11159
License (as archived)
CC0-1.0
Archived
2026-10-09T06:04:08.043Z
SHA-256 of the archived text
41b53f9b82a412354335a0de103a9b438ed5d4e19243a95e5cedbcca257d4ac9
Stable URL of this page
https://metraipolis.com/documento/dc_8be2fa3509c8

How to cite

Local Prototype Reconstruction for Text-Compatible Speech-to-LLM Bridge Pretraining — arXiv:2610.11159 — arXiv (en), https://arxiv.org/abs/2610.11159 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8be2fa3509c8 (Metraipolis archive dc_8be2fa3509c8, sha256 41b53f9b82a412354335a0de103a9b438ed5d4e19243a95e5cedbcca257d4ac9)
BibTeX
@misc{dc8be2fa3509c8,
  title = {Local Prototype Reconstruction for Text-Compatible Speech-to-LLM Bridge Pretraining — arXiv:2610.11159},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.11159},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8be2fa3509c8; sha256 41b53f9b82a412354335a0de103a9b438ed5d4e19243a95e5cedbcca257d4ac9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11159. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Speech-to-LLM systems often connect a frozen speech encoder to a frozen large language model (LLM) through a small trainable bridge. The bridge is usually treated as plumbing, but it in fact defines the geometry of the speech-to-LLM interface, and the pretraining objective decides whether that interface provides a reusable initialization for downstream tasks. We study a transferable bridge through two complementary properties: global alignment with the text side, and local lexical manifold compatibility, where bridge embeddings remain close to the frozen LLM's input-embedding neighbourhoods. We make this property measurable with a fixed, head-free, timestamp-free diagnostic that applies to any objective, and show that next-word prediction (NWP) and sentence-level contrastive pretraining do not fully capture token-level lexical compatibility. We then introduce Local Prototype Reconstruction (LPR), a lightweight training-only regularizer that requires each aligned bridge token to be reconstructable from a small neighbourhood of frozen LLM token embeddings, with a hard single-prototype anchor as its limiting case. On multilingual ASR and speech translation, LPR improves transfer, with the largest gains on translation and low-resource adaptation. Crucially, our independent diagnostic correlates with downstream gains across objectives, suggesting that lexical manifold compatibility is predictive of reusability for speech-to-LLM bridges.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Sound (cs.SD).
Pagina del lavoro: https://arxiv.org/abs/2610.11159 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:15.119Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).