Metraipolis › archive › dc_0cdef9fef79d

Binaural Audio-Visual Instance Segmentation — arXiv:2609.32180

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32180
License (as archived)
CC0-1.0
Archived
2026-10-07T23:50:45.712Z
SHA-256 of the archived text
f56f98d5c82d4ffe2de4690b645d8b7fad1bba7699b53eef1675e0f0a86df371
Stable URL of this page
https://metraipolis.com/documento/dc_0cdef9fef79d

How to cite

Binaural Audio-Visual Instance Segmentation — arXiv:2609.32180 — arXiv (en), https://arxiv.org/abs/2609.32180 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0cdef9fef79d (Metraipolis archive dc_0cdef9fef79d, sha256 f56f98d5c82d4ffe2de4690b645d8b7fad1bba7699b53eef1675e0f0a86df371)
BibTeX
@misc{dc0cdef9fef79d,
  title = {Binaural Audio-Visual Instance Segmentation — arXiv:2609.32180},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32180},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0cdef9fef79d; sha256 f56f98d5c82d4ffe2de4690b645d8b7fad1bba7699b53eef1675e0f0a86df371},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32180. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Audio-visual segmentation (AVS) aims to segment sounding objects at the pixel level by integrating auditory and visual cues. However, existing methods are predominantly developed under the monaural setting and primarily rely on cross-modal semantic correspondence, which limits their ability to distinguish visually similar instances of the same semantic class. In contrast, humans naturally exploit binaural hearing, where interaural differences and direction-dependent acoustic filtering introduced by the head and pinnae provide physically grounded spatial cues for accurate sound source localization. Motivated by this observation, we introduce binaural audio-visual instance segmentation (BiAVIS), a new task that leverages synchronized binaural audio and video frames to segment sounding instances. To advance research on this task, we establish two benchmarks by manually annotating an existing binaural audio-visual dataset and collecting a new real-world dataset, BiAVIS-Bench, in more challenging and diverse scenarios. We further propose a BiAVIS model, which leverages an audio-only sound source localization network to learn spatial and semantic priors for sounding instances from binaural audio. A query-level audio-visual fusion strategy is subsequently introduced to inject these informative priors into the instance segmentation decoder. Extensive experiments conducted on the two proposed benchmarks demonstrate the superior performance of the BiAVIS model over previous monaural AVS methods, especially in resolving instance-level intra-class ambiguity. On the more challenging BiAVIS-Bench, the proposed BiAVIS model outperforms the best-performing monaural baselines by 17.22\% in mAP and 7.71\% in FSLA, respectively.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Sound (cs.SD); Audio and Speech Processing (eess.AS).
Pagina del lavoro: https://arxiv.org/abs/2609.32180 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T23:29:26.975Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).