Metraipolis › archive › dc_46fa0ab54abc

Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories — arXiv:2609.37169

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.37169
License (as archived)
CC0-1.0
Archived
2026-10-07T16:26:57.934Z
SHA-256 of the archived text
5bd5e2e3fd0bad9b9d37936b6c80a73d95be8982db1e5be7335a88026ddf282e
Stable URL of this page
https://metraipolis.com/documento/dc_46fa0ab54abc

How to cite

Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories — arXiv:2609.37169 — arXiv (en), https://arxiv.org/abs/2609.37169 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_46fa0ab54abc (Metraipolis archive dc_46fa0ab54abc, sha256 5bd5e2e3fd0bad9b9d37936b6c80a73d95be8982db1e5be7335a88026ddf282e)
BibTeX
@misc{dc46fa0ab54abc,
  title = {Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories — arXiv:2609.37169},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37169},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_46fa0ab54abc; sha256 5bd5e2e3fd0bad9b9d37936b6c80a73d95be8982db1e5be7335a88026ddf282e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37169. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since additional serial compute yields little further downstream improvement and can even degrade some capabilities, which places a practical ceiling on how much compute mid-training absorbs. We revisit how this compute should be allocated to a single run or multiple similar optimizations. We find that branches forked from a shared checkpoint under various controlled recipe reaches measurably different regions of parameter space, and establish a form of compatible diversity that extending one run cannot supply. Therefore, we introduce Trajectory Soup, which distributes a mid-training budget over several independent branches, and consolidates strongest checkpoints selected on validation through intra- and inter-trajectory averaging into a single model. A local bias and variance analysis separates the two averaging levels, showing that inter-trajectory averaging removes residual error beyond the reach of averaging within a trajectory, while checkpoint selection carries a bias that bounds how many checkpoints are worth merging. Across model scales, learning-rate schedules, token budgets, and trajectory counts, Trajectory Soup improves aggregate downstream performance over the strongest single-trajectory average under matched budgets and keeps improving as budgets expand, with the advantage preserved after an identical post-training pipeline. These results position trajectory allocation and merging as a practical way to extend the compute-scaling frontier of mid-training beyond serial saturation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.37169 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.951Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).