Metraipolis › archive › dc_ab5acecbd2c5
Residual-Stream Burden Shapes Representation Learning in Diffusion Transformers — arXiv:2609.33895
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33895
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T22:40:15.579Z
- SHA-256 of the archived text
61260f5ba7d7450f00fe92a5f35fb7987bb8fd322f0bef5b44b2a50cda3c25e8- Stable URL of this page
https://metraipolis.com/documento/dc_ab5acecbd2c5
How to cite
Residual-Stream Burden Shapes Representation Learning in Diffusion Transformers — arXiv:2609.33895 — arXiv (en), https://arxiv.org/abs/2609.33895 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ab5acecbd2c5 (Metraipolis archive dc_ab5acecbd2c5, sha256 61260f5ba7d7450f00fe92a5f35fb7987bb8fd322f0bef5b44b2a50cda3c25e8)
BibTeX
@misc{dcab5acecbd2c5,
title = {Residual-Stream Burden Shapes Representation Learning in Diffusion Transformers — arXiv:2609.33895},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33895},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ab5acecbd2c5; sha256 61260f5ba7d7450f00fe92a5f35fb7987bb8fd322f0bef5b44b2a50cda3c25e8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33895. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
In diffusion-based generation, a neural network can be trained to predict the clean data, the noise, or the velocity from a noisy input. These prediction targets are interconvertible and describe the same generative process, yet plain Diffusion Transformers operating on large pixel patches succeed with clean prediction and fail with noise or velocity prediction. We argue that this asymmetry arises because noisy targets require the residual stream to preserve noise-dependent input variation through depth for the final readout, forcing subsequent layers to compute on noisy representations. A spectrally concentrated clean target imposes a lighter demand, leaving greater freedom to organize hidden representations for subsequent computation. We call this preservation requirement *residual-stream burden* and show how it shapes representation learning in Diffusion Transformers. Controlled experiments indicate that the exploitable structure is spectral concentration in patch space and that the bandwidth of the persistent residual state is a key resource for noisy prediction. We further show that this account is consistent with recent decoupled pixel-space architectures, whose diverse designs all reduce the residual-stream burden on the main pathway. To examine this understanding from a complementary direction, we expand and reorganize the residual-stream bandwidth directly, introducing Spatially Indexed Hyper-Connections (SiHC) that reach FID 1.71 on ImageNet $256^2$. Together, these results identify residual-stream burden as a mechanism through which prediction targets and architecture jointly shape representation learning in Diffusion Transformers. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.33895 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T22:14:28.794Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).