Metraipolis › archive › dc_a2dfa0265d83
Neural Spectral Capacity with Residual Context for Heterogeneous Resource Allocation within Transformers — arXiv:2610.30823
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.30823
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-23T15:15:07.788Z
- SHA-256 of the archived text
1502479bc5def26635f9e554529d3d30747f2178a6e57fc0740769801544de2f- Stable URL of this page
https://metraipolis.com/documento/dc_a2dfa0265d83
How to cite
Neural Spectral Capacity with Residual Context for Heterogeneous Resource Allocation within Transformers — arXiv:2610.30823 — arXiv (en), https://arxiv.org/abs/2610.30823 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a2dfa0265d83 (Metraipolis archive dc_a2dfa0265d83, sha256 1502479bc5def26635f9e554529d3d30747f2178a6e57fc0740769801544de2f)
BibTeX
@misc{dca2dfa0265d83,
title = {Neural Spectral Capacity with Residual Context for Heterogeneous Resource Allocation within Transformers — arXiv:2610.30823},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.30823},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a2dfa0265d83; sha256 1502479bc5def26635f9e554529d3d30747f2178a6e57fc0740769801544de2f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30823. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Predicting relative trained performance from architectural specifications can guide design when training costs limit experimentation. Neural Spectral Capacity (NSC) provides such a heuristic, but scores projections independently and assigns identical scores to compatible reorderings of the same layer configurations. We introduce Residual-Aware Neural Spectral Capacity (ResNSC) for Transformers, using a specification-derived statistical reference for accumulated residual updates to determine the input and residual-state statistics in each projection's capacity calculation, while retaining additive evaluation without data or instantiated weights. When residual widths vary, earlier choices affect later scores, so we augment NSC's allocator (NSC-DP) with residual history beyond its layer-and-budget state. ResNSC-DP jointly allocates residual and feed-forward widths under a resource budget, with certified bounds on the best numerically evaluated ResNSC score within a specified finite candidate set. At the 121M and 374M pretraining scales, selected architectures achieve lower mean perplexity than uniform, variable-width, and tapered allocations on FineWeb-Edu, WikiText-2, and LAMBADA at comparable parameter budgets. In inherited-weight LLaMA-7B pruning using the LoNAS supernet, ResNSC-DP improves eight-task average accuracy over NSC by 3.28--5.46 percentage points across eight matched parameter budgets. Paired reversals preserve the projection shapes and NSC scores yet worsen performance in both settings, exposing performance differences hidden by position-independent scoring. The code for this work is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.30823 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-23T15:09:08.556Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).