Metraipolis › archive › dc_e60fd8f84db5

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence — arXiv:2608.10720

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.10720
License (as archived)
CC0-1.0
Archived
2026-10-19T06:08:34.408Z
SHA-256 of the archived text
779b9e06ba277f6be873dab9ca014aecf9baf2b93c50232e8d7bbb791c742773
Stable URL of this page
https://metraipolis.com/documento/dc_e60fd8f84db5

How to cite

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence — arXiv:2608.10720 — arXiv (en), https://arxiv.org/abs/2608.10720 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e60fd8f84db5 (Metraipolis archive dc_e60fd8f84db5, sha256 779b9e06ba277f6be873dab9ca014aecf9baf2b93c50232e8d7bbb791c742773)
BibTeX
@misc{dce60fd8f84db5,
  title = {Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence — arXiv:2608.10720},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.10720},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e60fd8f84db5; sha256 779b9e06ba277f6be873dab9ca014aecf9baf2b93c50232e8d7bbb791c742773},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10720. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, but a spoken answer still leaves the agent visually absent. We introduce \textbf{Ex-Omni-2D}, a framework that answers a multimodal query with coordinated text, personalized speech, and reference-conditioned video. The dialogue model first writes a structured \textit{Visual Thought Plan} (VTP) for scene, emotion, and motion, then generates the response text and multi-codebook speech units. These speech units are decoded into audio and aligned with video frames, giving the speech and avatar modules a common timing signal while allowing them to learn from different data sources. The video module is trained as a full-sequence Teacher conditioned on reference appearance, VTP semantics, and frame-aligned speech units. We further explore to distill it into a few-step block-causal \emph{Streaming Student}; its Prefix Streaming mechanism carries the previous clean latent into the next chunk and is analyzed as a partial mitigation for late-chunk subject drift. At $400\times720$/$720\times400$, the four-step four-GPU Student provides incremental output with lower startup latency than the full-sequence Teacher.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.10720 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:57:28.292Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).