Metraipolis › archive › dc_0694126defed

Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages — arXiv:2608.29239

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.29239
License (as archived)
CC0-1.0
Archived
2026-10-17T08:06:54.361Z
SHA-256 of the archived text
f83471453d842e405466b23dc6337072e313d9501801d5528998ec83acfa721e
Stable URL of this page
https://metraipolis.com/documento/dc_0694126defed

How to cite

Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages — arXiv:2608.29239 — arXiv (en), https://arxiv.org/abs/2608.29239 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0694126defed (Metraipolis archive dc_0694126defed, sha256 f83471453d842e405466b23dc6337072e313d9501801d5528998ec83acfa721e)
BibTeX
@misc{dc0694126defed,
  title = {Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages — arXiv:2608.29239},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.29239},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0694126defed; sha256 f83471453d842e405466b23dc6337072e313d9501801d5528998ec83acfa721e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29239. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Low-resource ASR remains difficult because scarce transcripts provide limited supervised evidence for target-side generation. To address this gap, we propose SAMA-ASR, a lightweight adapter mechanism that augments the decoder with semantic anchors from auxiliary translations and an acoustic anchor from speech; in principle, the mechanism can be applied to similar encoder--decoder multitask speech models. Through cross-modal adaptation, SAMA-ASR conditions decoder states on translation-derived semantic embeddings and a speech embedding, combining utterance-level meaning with speech-grounded evidence before token prediction. At evaluation time, these semantic anchors can be generated automatically by an upstream speech-to-text translator rather than supplied as oracle translations. Experiments on two 30-hour datasets covering the low-resource Sinitic varieties Taiwanese Hokkien and Hakka show that SAMA-ASR improves over acoustic, prior prompt-based, and semantic-only translation-guided baselines and remains effective in practical automatic semantic-anchor settings; translator-capacity analyses show that useful semantic anchors can be produced by a compact ST model.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Sound (cs.SD); Audio and Speech Processing (eess.AS).
Pagina del lavoro: https://arxiv.org/abs/2608.29239 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:47:52.681Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).