Metraipolis › archive › dc_e67256b7eccd
Teaching LLMs to Hear Who Spoke What: Metadata-Supervised Pretraining for Encoder-Free Speech-LLMs — arXiv:2610.01695
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.01695
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T12:38:40.479Z
- SHA-256 of the archived text
3f64dd62a9216401276d1ffc3a8e94353586573bf1990bb112dfeabd97500c9f- Stable URL of this page
https://metraipolis.com/documento/dc_e67256b7eccd
How to cite
Teaching LLMs to Hear Who Spoke What: Metadata-Supervised Pretraining for Encoder-Free Speech-LLMs — arXiv:2610.01695 — arXiv (en), https://arxiv.org/abs/2610.01695 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e67256b7eccd (Metraipolis archive dc_e67256b7eccd, sha256 3f64dd62a9216401276d1ffc3a8e94353586573bf1990bb112dfeabd97500c9f)
BibTeX
@misc{dce67256b7eccd,
title = {Teaching LLMs to Hear Who Spoke What: Metadata-Supervised Pretraining for Encoder-Free Speech-LLMs — arXiv:2610.01695},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.01695},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e67256b7eccd; sha256 3f64dd62a9216401276d1ffc3a8e94353586573bf1990bb112dfeabd97500c9f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01695. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Encoder-based speech large language models (Speech-LLMs) commonly employ pretrained speech encoders that prioritize linguistic content but may discard fine-grained acoustic cues essential for speaker discrimination and paralinguistic understanding. Encoder-free Speech-LLMs instead map Mel-spectrogram features directly into the LLM input space through lightweight embedding layers, enabling the LLM to learn from low-level acoustic features. However, systematic pretraining strategies for encoder-free Speech-LLMs remain underexplored, limiting their ability to compensate for the absence of large-scale pretrained speech encoders. We propose metadata-supervised pretraining (MSP), which leverages speech attributes such as speaker identity and emotion to develop speaker-discriminative and paralinguistic capabilities. We further introduce speaker-aware utterance composition (SAUC) to strengthen speaker discrimination and apply random span masking to regularize pretraining. We primarily evaluate our approach on joint ASR and speaker diarization in multi-speaker conversations, complemented by experiments on paralinguistic speech-understanding tasks. Under matched training-data conditions, our encoder-free model outperforms its randomly initialized encoder-based counterpart. With limited metadata-annotated data, it is competitive with models using speech encoders pretrained on substantially larger corpora, outperforming them in several settings. These results demonstrate the potential of encoder-free architectures for building native multimodal LLMs that acquire diverse speech capabilities. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in eess.AS). Categorie arXiv: Audio and Speech Processing (eess.AS). Pagina del lavoro: https://arxiv.org/abs/2610.01695 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T12:11:28.581Z dalla pagina https://arxiv.org/catchup/eess.AS/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).