Metraipolis › archive › dc_86b8aa6f02ca

BAT-CLIP: Trimodal Alignment of Brain, Audio and Text — arXiv:2609.31180

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.31180
License (as archived): CC0-1.0
Archived: 2026-10-08T01:39:47.550Z
SHA-256 of the archived text: 6d069e1868171ff6fc356ef1a7be226b73e30e7d3110bbed6e2ec4f61f143fe9
Stable URL of this page: https://metraipolis.com/documento/dc_86b8aa6f02ca

How to cite

BAT-CLIP: Trimodal Alignment of Brain, Audio and Text — arXiv:2609.31180 — arXiv (en), https://arxiv.org/abs/2609.31180 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_86b8aa6f02ca (Metraipolis archive dc_86b8aa6f02ca, sha256 6d069e1868171ff6fc356ef1a7be226b73e30e7d3110bbed6e2ec4f61f143fe9)

BibTeX

@misc{dc86b8aa6f02ca,
  title = {BAT-CLIP: Trimodal Alignment of Brain, Audio and Text — arXiv:2609.31180},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.31180},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_86b8aa6f02ca; sha256 6d069e1868171ff6fc356ef1a7be226b73e30e7d3110bbed6e2ec4f61f143fe9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31180. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Decoding and interpreting naturalistic speech from the brain increasingly relies on alignment to pretrained speech and language representation spaces. However, current CLIP-style brain-speech alignment ground neural activity to a single anchor modality-audio or text-despite the brain's inherently multimodal speech processing. This induces a trade-off: audio anchoring preserves temporal structure but weakens linguistic separability, while text anchoring captures semantics yet discards acoustic detail. We propose BAT-CLIP, the first CLIP-style trimodal alignment framework for iEEG that jointly aligns neural embeddings to both pretrained audio and text anchors in a shared, frozen audio-text manifold. On the naturalistic Podcast benchmark, BAT-CLIP yields more robust representations than bimodal CLIP baselines. We also highlight the importance of using self-supervised foundation models for CLIP training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.AI).
Categorie arXiv: Sound (cs.SD) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS).
Pagina del lavoro: https://arxiv.org/abs/2609.31180 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T01:12:14.552Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).