Metraipolis › archive › dc_5fef0ecc12a7

NepTTS-Bench: Native-Speaker Benchmarking and NepaliMOS for Nepali Text-to-Speech — arXiv:2611.03718

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.03718
License (as archived)
CC0-1.0
Archived
2026-11-04T18:35:36.549Z
SHA-256 of the archived text
2df168bb2ad72a07f7f776c1d71e68ccaf2f27523503af042d314769cf02747d
Stable URL of this page
https://metraipolis.com/documento/dc_5fef0ecc12a7

How to cite

NepTTS-Bench: Native-Speaker Benchmarking and NepaliMOS for Nepali Text-to-Speech — arXiv:2611.03718 — arXiv (en), https://arxiv.org/abs/2611.03718 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5fef0ecc12a7 (Metraipolis archive dc_5fef0ecc12a7, sha256 2df168bb2ad72a07f7f776c1d71e68ccaf2f27523503af042d314769cf02747d)
BibTeX
@misc{dc5fef0ecc12a7,
  title = {NepTTS-Bench: Native-Speaker Benchmarking and NepaliMOS for Nepali Text-to-Speech — arXiv:2611.03718},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03718},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5fef0ecc12a7; sha256 2df168bb2ad72a07f7f776c1d71e68ccaf2f27523503af042d314769cf02747d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03718. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Nepali TTS has reached a point where multiple systems are available, but there is no shared evaluation to compare them. NepTTS-Bench was created to address what existing automated and manual evaluation systems lack for Nepali TTS. A controlled benchmark with phonological grounding which contains 205 sentences (77 minimal pairs targeting aspiration, retroflex/dental contrasts, nasalization, gemination, and schwa deletion) which is rated by 180 speakers. There are 6,962 ratings for 10 TTS configurations and two human references. Along with this 7 ASR systems were evaluated for round-trip CER. SCOREQ was also run as an automated baseline. The key finding is that automated evaluation like SCOREQ prefers proprietary systems like ElevenLabs but native listeners prefer Piper. This is contributed by the fact that most of the systems not properly tuned for Nepali tend to show Hindi phonological behaviour and our minimal pairs are designed to surface such contrasts. We also present NepaliMOS predictor which is fine tuned on collected ratings. NepaliMOS achieves system-level rho = 0.90 on the dense panel but drops to utterance-level rho = 0.43 under leave-one-system-out evaluation. This is statistically significant and confirms that NepaliMOS captures genuine signal but cross-provider transfer is limited. A human calibration is recommended for new systems for which we release a Python evaluation package for practitioners to run the benchmark against new systems.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Audio and Speech Processing (eess.AS) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.03718 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:32:37.376Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).