Metraipolis › archive › dc_d0143bf070ce
Enhancing Neural Speech Coding with Semantic and Visual Cues — arXiv:2609.05076
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.05076
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T00:43:26.686Z
- SHA-256 of the archived text
28a2ff9e28bc15364a9ce5a5ab8652f26cd80bdd6bc3e99264da44239b881198- Stable URL of this page
https://metraipolis.com/documento/dc_d0143bf070ce
How to cite
Enhancing Neural Speech Coding with Semantic and Visual Cues — arXiv:2609.05076 — arXiv (en), https://arxiv.org/abs/2609.05076 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d0143bf070ce (Metraipolis archive dc_d0143bf070ce, sha256 28a2ff9e28bc15364a9ce5a5ab8652f26cd80bdd6bc3e99264da44239b881198)
BibTeX
@misc{dcd0143bf070ce,
title = {Enhancing Neural Speech Coding with Semantic and Visual Cues — arXiv:2609.05076},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.05076},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d0143bf070ce; sha256 28a2ff9e28bc15364a9ce5a5ab8652f26cd80bdd6bc3e99264da44239b881198},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05076. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
At low bitrates, neural speech codecs have limited capacity to encode all information needed for high-quality re construction, especially when relying solely on speech-derived representations. To address this limitation, this paper proposes a Semantic- and Visual-enhanced Speech Codec (SVSC), which in corporates semantic and visual cues into the neural speech coding process. Specifically, built upon a mainstream neural speech cod ing architecture, SVSC introduces a semantic encoding-decoding branch and an image analysis-synthesis branch. It fuses deep semantic features with visual cues through a cross-attention mech anism, forming an auxiliary high-level representation enriched with contextual and articulatory information. To handle different inference scenarios, SVSC introduces two information-injection strategies based on the availability of auxiliary semantic and vi sual cues. When such cues are available, the fusion mode directly incorporates the auxiliary representations into the speech coding branch through feature concatenation; otherwise, the distillation mode transfers auxiliary information into the speech coding branch through knowledge distillation during training, enabling speech-only inference without additional inputs. Experimental results validate the effectiveness of incorporating semantic and visual cues, improving the ViSQOL score of reconstructed speech from 3.86 to 4.01. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in eess.AS). Categorie arXiv: Audio and Speech Processing (eess.AS). Pagina del lavoro: https://arxiv.org/abs/2609.05076 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T00:21:40.227Z dalla pagina https://arxiv.org/catchup/eess.AS/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).