Metraipolis › archive › dc_87678cf04cf7

Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation — arXiv:2609.30517

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.30517
License (as archived)
CC0-1.0
Archived
2026-10-08T02:19:08.914Z
SHA-256 of the archived text
8addf5acdefba23aef213bc23d8fb9ac2242525cfc8a4db95de1a1f3d247d0d2
Stable URL of this page
https://metraipolis.com/documento/dc_87678cf04cf7

How to cite

Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation — arXiv:2609.30517 — arXiv (en), https://arxiv.org/abs/2609.30517 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_87678cf04cf7 (Metraipolis archive dc_87678cf04cf7, sha256 8addf5acdefba23aef213bc23d8fb9ac2242525cfc8a4db95de1a1f3d247d0d2)
BibTeX
@misc{dc87678cf04cf7,
  title = {Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation — arXiv:2609.30517},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.30517},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_87678cf04cf7; sha256 8addf5acdefba23aef213bc23d8fb9ac2242525cfc8a4db95de1a1f3d247d0d2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.30517. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent progress in speech-driven 3D facial animation has improved vertex-level reconstruction quality, but speech-consistent visible articulation remains difficult. This is because speech production follows structured and constrained articulators' coordination and the mapping from acoustics to motion is inherently one-to-many. Motivated by the structured patterns of visible articulation, we propose a novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion. To represent visible articulation with three directional articulatory motions, spreading, opening, and protrusion, we propose a Speech--Articulatory Memory (SAM) that captures the correspondence between speech and these motions under phonetic context through retrieval and decoding based on a key-value memory structure. Then, a Topology-aware Articulatory Composition (TAC) integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion. Experiments on VOCASET and TFHP show that our method achieves state-of-the-art performance on standard reconstruction metrics and improves visible articulatory distance and velocity errors for lip articulation, while a user study confirms clear preference in lip sync and realism.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.LG).
Categorie arXiv: Audio and Speech Processing (eess.AS) ; Graphics (cs.GR); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.30517 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T01:43:06.781Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).