Metraipolis › archive › dc_552917778cd3
NAVIR: Neuromorphic Audio-Visual Speech Recognition for Robust Human-Robot Interaction on Edge Hardware — arXiv:2609.24391
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.24391
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T07:07:44.420Z
- SHA-256 of the archived text
54adcf2e2f4f45959b18b68d1574fc0fa9210c1432c6d982a651f083ee894616- Stable URL of this page
https://metraipolis.com/documento/dc_552917778cd3
How to cite
NAVIR: Neuromorphic Audio-Visual Speech Recognition for Robust Human-Robot Interaction on Edge Hardware — arXiv:2609.24391 — arXiv (en), https://arxiv.org/abs/2609.24391 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_552917778cd3 (Metraipolis archive dc_552917778cd3, sha256 54adcf2e2f4f45959b18b68d1574fc0fa9210c1432c6d982a651f083ee894616)
BibTeX
@misc{dc552917778cd3,
title = {NAVIR: Neuromorphic Audio-Visual Speech Recognition for Robust Human-Robot Interaction on Edge Hardware — arXiv:2609.24391},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.24391},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_552917778cd3; sha256 54adcf2e2f4f45959b18b68d1574fc0fa9210c1432c6d982a651f083ee894616},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24391. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Voice-controlled interaction in industrial settings is hampered by acoustic noise, which severely degrades audio-only speech recognition. Audio-visual speech recognition (AVSR) addresses this by fusing lip-motion cues with the audio stream, but state-of-the-art pipelines rely on three-dimensional convolutions, recurrent units, and attention modules that exceed the budget of typical edge devices. We present NAVIR, an end-to-end AVSR system targeting the BrainChip Akida neuromorphic processor, which natively supports only sequential two-dimensional convolutional inference. The pipeline factorises spatial and temporal encoding into separate AkidaNet-based modules: a per-frame visual encoder, a temporal video encoder, and a spectrogram audio encoder, fused by a lightweight predictor head and decoded by constrained beam search. Models are trained with connectionist temporal classification on noise-augmented audio and then fine-tuned with quantization-aware training. On the GRID benchmark, the quantized audio-visual model reaches 14.0% word error rate (WER) under noise on the unseen-speaker split and 3.3% WER on the overlapped-speaker split, against 22.5% and 11.8% for audio-only baselines, and it attains 98.6% command accuracy at 1.5% WER on a task-specific industrial-command corpus. Operation-count analysis indicates a 13-fold energy advantage of the spiking formulation over its artificial neural network counterpart at 27.6% mean firing rate. On-board measurements show roughly 5-fold lower energy per inference than a Raspberry Pi central processing unit on the lip-reading model, and over 100-fold lower than a laptop graphics processing unit, while sustaining 14.5 inferences per second. To the best of our knowledge, this is the first complete multimodal AVSR pipeline running on neuromorphic hardware of this class. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.24391 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T06:58:09.408Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).