Metraipolis › archive › dc_9b7845a4e13a

Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping — arXiv:2610.32492

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.32492
License (as archived)
CC0-1.0
Archived
2026-10-26T19:53:50.577Z
SHA-256 of the archived text
2cf01e3de79be3d21701d7f8538b76b88be9e85bf5ae63d7649f093451024b7e
Stable URL of this page
https://metraipolis.com/documento/dc_9b7845a4e13a

How to cite

Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping — arXiv:2610.32492 — arXiv (en), https://arxiv.org/abs/2610.32492 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9b7845a4e13a (Metraipolis archive dc_9b7845a4e13a, sha256 2cf01e3de79be3d21701d7f8538b76b88be9e85bf5ae63d7649f093451024b7e)
BibTeX
@misc{dc9b7845a4e13a,
  title = {Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping — arXiv:2610.32492},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.32492},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9b7845a4e13a; sha256 2cf01e3de79be3d21701d7f8538b76b88be9e85bf5ae63d7649f093451024b7e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.32492. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substantial costs. Recent Looped Transformers enable flexible inference computation through parameter reuse, but maintaining strong performance across different inference budgets remains an open challenge. We introduce Accordion, a mixture-of-experts-based model that supports five inference budgets through fixed-prefix layer looping. Each budget follows a pre-defined execution path, while the minimum budget retains the unlooped model. Joint pretraining samples one budget per micro-batch and optimizes its next-token prediction loss without additional cross-budget consistency objectives. We further preserve these inference budgets through supervised fine-tuning and reinforcement learning, where parameter-group merging and budget-matched multi-teacher on-policy distillation integrate three domain experts into one model. After 3.6T tokens of pretraining, increasing inference depth improves Accordion-6B-A0.5B's five-category macro-average across 37 tasks by 3.01 percentage points, with the middle budget capturing 89% of this gain using half as many additional layer executions. The final model surpasses each domain expert on its corresponding task-and-budget average, while achieving strong instruction-following performance across all budgets and retaining depth-dependent gains in mathematical reasoning and code generation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.32492 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:45:48.393Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).