Metraipolis › archive › dc_ba2f6ed7fa23

Multimodal Emergency Vehicle Classification via Audio-Visual Transformers and Knowledge Distillation — arXiv:2609.16535

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.16535
License (as archived)
CC0-1.0
Archived
2026-10-15T21:52:12.446Z
SHA-256 of the archived text
9917f8c26846b10a08a359ba1383f03cd0763054cae732f05ed6da2377481fb0
Stable URL of this page
https://metraipolis.com/documento/dc_ba2f6ed7fa23

How to cite

Multimodal Emergency Vehicle Classification via Audio-Visual Transformers and Knowledge Distillation — arXiv:2609.16535 — arXiv (en), https://arxiv.org/abs/2609.16535 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ba2f6ed7fa23 (Metraipolis archive dc_ba2f6ed7fa23, sha256 9917f8c26846b10a08a359ba1383f03cd0763054cae732f05ed6da2377481fb0)
BibTeX
@misc{dcba2f6ed7fa23,
  title = {Multimodal Emergency Vehicle Classification via Audio-Visual Transformers and Knowledge Distillation — arXiv:2609.16535},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.16535},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ba2f6ed7fa23; sha256 9917f8c26846b10a08a359ba1383f03cd0763054cae732f05ed6da2377481fb0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16535. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Emergency vehicle detection in autonomous driving is a safety-critical perception task that demands robustness under diverse and adverse real-world conditions. Existing approaches rely on a single modality, either audio or video, which leads to systematic failure when that modality is degraded: microphone-based systems fail in noisy urban environments, and camera-based systems fail at night or under occlusion. This report presents AVNet, a multimodal audio-visual transformer that classifies emergency vehicles (ambulance, fire engine, police car) and road background using both audio and video, while gracefully handling the absence of either modality at inference time. AVNet introduces three key contributions: (1) a temporally aligned cross-modal fusion module that performs second-level cross-attention between audio spectrogram tokens and video frame tokens, exploiting their exact temporal correspondence without any learned alignment mechanism; (2) learned null embeddings that substitute for missing modality tokens, enabling a single unified model to operate in audio-only, video-only, or joint audio-visual mode without retraining; and (3) a knowledge distillation training strategy in which specialist unimodal teacher models transfer inter-class dark knowledge into the multimodal student fusion branch via soft probability targets. Evaluated on 281 clips from the Google AudioSet dataset, AVNet achieves 66.6% overall accuracy in audio-visual mode, outperforming the audio-only branch by +10.4% and the video-only branch by +15.0%. The largest per-class gain is observed for the hardest class, Ambulance, where fusion achieves +29.5% over either unimodal branch alone, demonstrating that the two modalities provide complementary information that the aligned cross attention mechanism successfully exploits.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.MM).
Categorie arXiv: Multimedia (cs.MM) ; Sound (cs.SD).
Pagina del lavoro: https://arxiv.org/abs/2609.16535 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T21:36:31.121Z dalla pagina https://arxiv.org/catchup/cs.MM/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).