Metraipolis › archive › dc_e1fa7de6a03f

A Distributional Optimisation Perspective on Combining Models in Deep Learning — arXiv:2609.24328

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.24328
License (as archived)
CC0-1.0
Archived
2026-10-09T07:07:40.194Z
SHA-256 of the archived text
79f2c4922e24e87de8e930da905453d6aad955e7e740f675f0b186eb9c5acbc5
Stable URL of this page
https://metraipolis.com/documento/dc_e1fa7de6a03f

How to cite

A Distributional Optimisation Perspective on Combining Models in Deep Learning — arXiv:2609.24328 — arXiv (en), https://arxiv.org/abs/2609.24328 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e1fa7de6a03f (Metraipolis archive dc_e1fa7de6a03f, sha256 79f2c4922e24e87de8e930da905453d6aad955e7e740f675f0b186eb9c5acbc5)
BibTeX
@misc{dce1fa7de6a03f,
  title = {A Distributional Optimisation Perspective on Combining Models in Deep Learning — arXiv:2609.24328},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.24328},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e1fa7de6a03f; sha256 79f2c4922e24e87de8e930da905453d6aad955e7e740f675f0b186eb9c5acbc5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24328. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Combining predictions from different models can improve performance at machine learning tasks, but the training of the individual models and the rule used to combine them are typically chosen separately, and by ad hoc means. Recent advances in distributional optimisation (i.e. where the optimisation occurs over the set of probability distributions) offer an opportunity for principled joint training, viewing the collection of models as a discrete distribution whose support points are to be optimised, but the potential of these methods is not well-understood. In this paper we (1) cast two standard combination strategies - ensembles and low-rank adapter averaging - as entropy-regularised distributional optimisation, observing that the resulting objective is convex in the ensemble case but not in the adapter-averaging case, so that existing convergence guarantees for mean field Langevin dynamics transfer only to the former; (2) assess existing and novel algorithms for this task, including a functional variant of variational gradient descent; and (3) report an empirical study spanning synthetic classification tasks and fine-tuning of large language models on a commonsense reasoning benchmark.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.24328 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:58:09.396Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).