Metraipolis › archive › dc_47b7c154dc68

Temporal-Attention Head Specialization During Video Diffusion Training — arXiv:2609.31654

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.31654
License (as archived)
CC0-1.0
Archived
2026-10-07T20:59:15.906Z
SHA-256 of the archived text
8275414398c452c4370837633762e8a2c83b40a35d28a96ab4a906f0429c0bf6
Stable URL of this page
https://metraipolis.com/documento/dc_47b7c154dc68

How to cite

Temporal-Attention Head Specialization During Video Diffusion Training — arXiv:2609.31654 — arXiv (en), https://arxiv.org/abs/2609.31654 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_47b7c154dc68 (Metraipolis archive dc_47b7c154dc68, sha256 8275414398c452c4370837633762e8a2c83b40a35d28a96ab4a906f0429c0bf6)
BibTeX
@misc{dc47b7c154dc68,
  title = {Temporal-Attention Head Specialization During Video Diffusion Training — arXiv:2609.31654},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.31654},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_47b7c154dc68; sha256 8275414398c452c4370837633762e8a2c83b40a35d28a96ab4a906f0429c0bf6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31654. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Video diffusion transformers depend on temporal attention to coordinate information across frames, yet nearly everything known about this mechanism comes from analyzing trained models, so when and where temporal-attention structure forms during training remains poorly characterized. Population averages can also hide it, since a few specializing heads and a diffusing majority cancel in the mean. We therefore conduct a checkpoint-resolved census of every temporal-attention head across nine Open-Sora STDiT training runs spanning three model scales (306M to 1.03B parameters), scoring each head with an entropy-normalized measure of cross-frame attention concentration (CFAC) under a preregistered change-point and effect-size selection rule. The census reveals the sparse picture that averages obscure. Aggregate CFAC is flat or decreasing in every run, while a small minority of heads, roughly 4--13% in full-grid runs, develops pronounced concentration. Across seeds, the reproducible signal is positional but block-level. Selected heads repeatedly arise in the first temporal block, whereas individual head coordinates do not reproduce once block membership is accounted for. Among the analyzed 760M selected heads, attention maps converge to a small repertoire of local frame-routing motifs, self-frame diagonals and adjacent-frame bands, even when the responsible coordinates differ across runs. Correlation and ablation analyses do not establish a causal link to generated video quality, and we bound our claims accordingly. Beyond this STDiT family, the study contributes a transferable methodology. Checkpoint-resolved, per-head analysis under fixed selection rules can expose sparse temporal organization in other factorized video diffusion transformers and, with adapted routing metrics, in joint spatio-temporal architectures.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.31654 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.999Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).