Metraipolis › archive › dc_2300a626f224

Identity-Faithful Audio-Visual Target Speaker Extraction with REAL-2MIX and VOXBLINK2-AVSE — arXiv:2608.03964

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.03964
License (as archived): CC0-1.0
Archived: 2026-10-31T13:28:53.346Z
SHA-256 of the archived text: 5a595cd9618465dba7646584fa9fa39824df84cdd816f46dcef0ab29179ed91f
Stable URL of this page: https://metraipolis.com/documento/dc_2300a626f224

How to cite

Identity-Faithful Audio-Visual Target Speaker Extraction with REAL-2MIX and VOXBLINK2-AVSE — arXiv:2608.03964 — arXiv (en), https://arxiv.org/abs/2608.03964 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2300a626f224 (Metraipolis archive dc_2300a626f224, sha256 5a595cd9618465dba7646584fa9fa39824df84cdd816f46dcef0ab29179ed91f)

BibTeX

@misc{dc2300a626f224,
  title = {Identity-Faithful Audio-Visual Target Speaker Extraction with REAL-2MIX and VOXBLINK2-AVSE — arXiv:2608.03964},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03964},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2300a626f224; sha256 5a595cd9618465dba7646584fa9fa39824df84cdd816f46dcef0ab29179ed91f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03964. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice. We introduce REAL-2MIX, a Mandarin AV-TSE benchmark of jointly recorded real two-speaker mixtures with synchronized multi-view video. Each scene also contains preceding A-only and B-only stages that provide in-scene speaker references. It contains 77 scenes and 7,598 clips (11.84 hours), including 6,042 dual-annotated mixtures. After processing there leave 6,038 evaluable mixtures and 12,076 target-speaker rows. We additionally curate VOXBLINK2-AVSE from VoxBlink2, comprising 250,828 synchronized audio--lip-ROI pairs from 28,421 identities and 766.17 hours of speech. Our extractor uses frozen, 1,280-dimensional projected AV-HuBERT features, target-conditioned training, and layer-wise feature modulation. We jointly evaluate content with Qwen3-ASR-1.7B CER and target identity with WeSpeaker ResNet34 plus Overlapped Speech Detection (OSD). On the complete manifest, the best archived checkpoint obtains 0.2261 CER, 82.22% strict output correctness, and 69.53% both-output strict success.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in eess.AS).
Categorie arXiv: Audio and Speech Processing (eess.AS).
Pagina del lavoro: https://arxiv.org/abs/2608.03964 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T13:15:28.102Z dalla pagina https://arxiv.org/catchup/eess.AS/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).