Metraipolis › archive › dc_bbf2a3afcb40
Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling — arXiv:2610.19559
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.19559
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T15:03:49.518Z
- SHA-256 of the archived text
18deb7bda06d8f4307d92733a1fea270e0ed0ff7be50feb0a7677d4b6e1d4f19- Stable URL of this page
https://metraipolis.com/documento/dc_bbf2a3afcb40
How to cite
Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling — arXiv:2610.19559 — arXiv (en), https://arxiv.org/abs/2610.19559 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bbf2a3afcb40 (Metraipolis archive dc_bbf2a3afcb40, sha256 18deb7bda06d8f4307d92733a1fea270e0ed0ff7be50feb0a7677d4b6e1d4f19)
BibTeX
@misc{dcbbf2a3afcb40,
title = {Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling — arXiv:2610.19559},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.19559},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bbf2a3afcb40; sha256 18deb7bda06d8f4307d92733a1fea270e0ed0ff7be50feb0a7677d4b6e1d4f19},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19559. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this work, we propose FLARE, a framework for robust visual speech representation learning via cross-modal distillation and joint face-lip modeling. FLARE leverages semantically rich Whisper representations as the acoustic teacher signal, converting them into discrete distillation targets through a random-projection quantizer and pretraining the model with a masked prediction objective. To capture complementary visual information, FLARE employs a dual-stream architecture that jointly processes lip region-of-interest (ROI) and full-face inputs, where the lip stream captures finegrained articulatory motion and the face stream encodes broader facial dynamics. During pretraining, auxiliary audio input is incorporated to facilitate cross-modal alignment, and modality dropout is applied to encourage robust visual-only representations. For downstream lipreading, FLARE serves as a visual encoder coupled with either a lightweight Transformer decoder or an LLM-based decoder. Evaluated on LRS2, LRS3, and the out-of-domain WildVSR benchmark, FLARE achieves strong lipreading performance, with word error rates of 11.7%, 13.2%, and 34.3%, respectively. A progressive design study confirms the contribution of each major component. Further analyses demonstrate that joint face-lip modeling improves robustness under degraded video resolution and large head pose variation, while yielding more phoneme-discriminative visual representations. Code and pretrained models are available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv eess.AS, nuovi lavori del giorno). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Audio and Speech Processing (eess.AS). Pagina del lavoro: https://arxiv.org/abs/2610.19559 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T14:07:38.583Z dalla pagina https://arxiv.org/list/eess.AS/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).