Metraipolis › archive › dc_156bd3d44be4

Successive Capacity Growth: Task-Complexity-Driven Width and Depth Expansion for Vision Transformer Encoders in JEPA World Models — arXiv:2608.27367

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27367
License (as archived)
CC0-1.0
Archived
2026-10-17T14:31:01.167Z
SHA-256 of the archived text
276f0e480496a8b7a1ded41676a85fbfa58bd7c3a5c9d5807ed88a75ab744823
Stable URL of this page
https://metraipolis.com/documento/dc_156bd3d44be4

How to cite

Successive Capacity Growth: Task-Complexity-Driven Width and Depth Expansion for Vision Transformer Encoders in JEPA World Models — arXiv:2608.27367 — arXiv (en), https://arxiv.org/abs/2608.27367 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_156bd3d44be4 (Metraipolis archive dc_156bd3d44be4, sha256 276f0e480496a8b7a1ded41676a85fbfa58bd7c3a5c9d5807ed88a75ab744823)
BibTeX
@misc{dc156bd3d44be4,
  title = {Successive Capacity Growth: Task-Complexity-Driven Width and Depth Expansion for Vision Transformer Encoders in JEPA World Models — arXiv:2608.27367},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27367},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_156bd3d44be4; sha256 276f0e480496a8b7a1ded41676a85fbfa58bd7c3a5c9d5807ed88a75ab744823},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27367. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Joint-Embedding Predictive Architectures (JEPAs) for world modeling typically employ fixed-size Vision Transformer encoders that are over-provisioned for simple tasks and under-provisioned for complex ones, with significant redundancy across attention heads. We propose Successive Capacity Growth (SCG), a method that starts from a minimal encoder (1 head, 2 layers, 283K parameters) and grows incrementally in width (adding attention heads for low-level semantic capacity) or depth (adding transformer blocks for higher-order semantic abstraction), driven by a task-agnostic test-and-verify mechanism that exploits function-preserving expansion to safely trial architectural changes and roll back if they do not improve prediction loss. The Sketched Isotropic Gaussian Regularizer (SIGReg) ensures that all learned semantic dimensions remain statistically independent and aligned with the predictive objective, preventing collapse even as the architecture grows. On a 60-dimensional multi-object dynamics task, SCG naturally triggers depth expansion, improving prediction loss by 20.3% over the fixed small baseline with 56 times greater parameter efficiency than scaling to the fixed large model; on a 2D navigation task, a single width expansion yields even an 23% improvement over the fixed large model. Across all three tested environments of increasing complexity, the adaptive encoder matches or exceeds the fixed small baseline, with zero false-positive expansions and bit-exact function preservation (ratio = 1.0, absolute difference = 0.0). The take-away is that JEPA world model encoders need not be pre-allocated at maximum capacity - they can grow successively as the task demands, achieving significant compute and data efficiency while maintaining representation quality.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.AI).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.27367 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T13:16:53.713Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).