Metraipolis › archive › dc_6cd4864dd50f

Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling — arXiv:2608.02829

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.02829
License (as archived)
CC0-1.0
Archived
2026-10-31T11:13:22.087Z
SHA-256 of the archived text
954db58c35362800429559d74fe16e0aa622ed0111a9300f4fd811b4157d2149
Stable URL of this page
https://metraipolis.com/documento/dc_6cd4864dd50f

How to cite

Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling — arXiv:2608.02829 — arXiv (en), https://arxiv.org/abs/2608.02829 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6cd4864dd50f (Metraipolis archive dc_6cd4864dd50f, sha256 954db58c35362800429559d74fe16e0aa622ed0111a9300f4fd811b4157d2149)
BibTeX
@misc{dc6cd4864dd50f,
  title = {Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling — arXiv:2608.02829},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.02829},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6cd4864dd50f; sha256 954db58c35362800429559d74fe16e0aa622ed0111a9300f4fd811b4157d2149},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.02829. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Model families are trained size by size. Can a pretrained large model instead be converted into a smaller sibling? We characterize the 1.4B->410M conversion in Pythia end to end. Representations align strongly across sizes (ridge R^2=0.84); parameters align weakly. Dense weight projection is destructive; a bit-exact control places the fault in basis mixing, which breaks rotary, per-head, GELU, and LayerNorm structure. Residuals after the best-fit linear operator carry no learnable or transferable signal under shuffle controls, so conversion value lives in initialization. Matched-budget continued pre-training separates two independent levers, least-squares compensation (function lever, best zero-shot) and variance-preserving rescale (dynamics lever, best endpoints), with a placement rule read off the architecture: compensation is well-posed exactly where no normalization sits between cut and read; norm-fronted paths take rescale. At 30M tokens compensation beats the best subcloning variant on a width-reduced pair (84.6+-2.1 vs. 90.7+-3.8, 6/6 seeds) and a held-out depth-reduced pair (109.3 vs. 117.9, 3/3 seeds); the two reach parity at 33x the budget, and at ~5x the donor scale the levers anti-synergize. A second family, Qwen3-1.7B->0.6B (RMSNorm, SwiGLU, grouped-query attention), takes the placement rule unchanged but reverses a 2x2 crossing the re-fit with the activation statistic it reads: on Pythia the re-fit carries the gain and the statistic adds 2.3 perplexity; on Qwen3, with far more heterogeneous MLP unit energies, the statistic carries 66 points and the re-fit is a null on top. Informed selection + compensation + rescale is the best arm in both families, 19x and 23x lower perplexity than from scratch at matched budget (2.94 and 3.13 nats), and beats structured pruning plus distillation. Code, checkpoints, and the frozen eval corpus are released.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.02829 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:02:13.451Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).