Metraipolis › archive › dc_ddd77ef7f330

FARO: Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation — arXiv:2610.40858

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.40858
License (as archived)
CC0-1.0
Archived
2026-11-03T07:25:49.944Z
SHA-256 of the archived text
40d73b9b32a9cefa3bdd8ab77452f7ff261fc125970acb6a61401e881a5f89e8
Stable URL of this page
https://metraipolis.com/documento/dc_ddd77ef7f330

How to cite

FARO: Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation — arXiv:2610.40858 — arXiv (en), https://arxiv.org/abs/2610.40858 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ddd77ef7f330 (Metraipolis archive dc_ddd77ef7f330, sha256 40d73b9b32a9cefa3bdd8ab77452f7ff261fc125970acb6a61401e881a5f89e8)
BibTeX
@misc{dcddd77ef7f330,
  title = {FARO: Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation — arXiv:2610.40858},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40858},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ddd77ef7f330; sha256 40d73b9b32a9cefa3bdd8ab77452f7ff261fc125970acb6a61401e881a5f89e8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40858. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Training a large language model is compute-bound, and current recipes divide the work between optimizers by hand: orthogonalized momentum for attention and MLP matrices, Adam-style updates for the rest. This division rests on human priors about which layer type deserves which geometry, and it fixes a single global update budget that every parameter block shares. We introduce FARO, a function-space adaptive optimizer that makes the partition of parameters into blocks the unit of decision. FARO gives each block a geometry by its realized counterfactual return, then freezes the routes and reallocates the frozen per-block budget by cross-block equal risk contribution (ERC), the risk-parity principle from portfolio theory. On a GPT-style model trained on FineWeb, the framework gives two results. First, its FARO-Adaptive arm reaches 3.2780 nats at the full schedule, matching NorMuonAndAdam (3.2794), the best hand-designed recipe: it recovers the human prior without hand-setting per-layer geometry. Second, FARO-ERC improves on that arm: it lowers the clean-data loss at every one of 6 seeds, by 0.0136 nats on average, a modest but robust gain. Its sign test, paired test, and confidence interval all agree, and it exceeds the measured run-to-run noise at no added time or memory. It works by trimming the riskiest blocks rather than expanding the safest, and it holds up under a corrupted stream, still lowering the mean loss and winning the seed majority. The source code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.40858 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:20:03.764Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).