Metraipolis › archive › dc_da97d0bc875e

Trapped by Their Own Rollouts: Understanding Aggregation--Rollout Feedback in Federated On-Policy Distillation — arXiv:2609.32573

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.32573
License (as archived): CC0-1.0
Archived: 2026-10-07T22:12:19.776Z
SHA-256 of the archived text: e0be8873f1780984dae9eb6de0a2cce02707b0824c7d1dae52b151a0bc31b7ec
Stable URL of this page: https://metraipolis.com/documento/dc_da97d0bc875e

How to cite

Trapped by Their Own Rollouts: Understanding Aggregation--Rollout Feedback in Federated On-Policy Distillation — arXiv:2609.32573 — arXiv (en), https://arxiv.org/abs/2609.32573 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_da97d0bc875e (Metraipolis archive dc_da97d0bc875e, sha256 e0be8873f1780984dae9eb6de0a2cce02707b0824c7d1dae52b151a0bc31b7ec)

BibTeX

@misc{dcda97d0bc875e,
  title = {Trapped by Their Own Rollouts: Understanding Aggregation--Rollout Feedback in Federated On-Policy Distillation — arXiv:2609.32573},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32573},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_da97d0bc875e; sha256 e0be8873f1780984dae9eb6de0a2cce02707b0824c7d1dae52b151a0bc31b7ec},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32573. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) is a promising approach to language-model adaptation, aligning teacher supervision with the student's own generated trajectories. When adaptation prompts are distributed across clients, can this process benefit from federated collaboration? We study federated OPD and find that substantial collaboration gains can be obscured by learning-rate sensitivity: FedAvg can perform no better than independent local training at a small learning rate, yet recover a clear advantage at a larger rate. We explain this phenomenon through the student's dual role as learner and generator of future training data. An aggregation-induced optimization lag can delay access to useful teacher supervision, which in turn slows subsequent learning. Our theory establishes this aggregation--rollout feedback in a solvable model with a common optimum and stable updates, and identifies two coupled roles of learning rate: learning from current supervision and reaching future supervision. Guided by this analysis, we propose FedTOPS (Federated Teacher-guided On-Policy Scaling), which reuses teacher feedback on current trajectories to adapt the FedAvg update magnitude under clientwise predictive-change constraints. Across six mathematical reasoning benchmarks, FedTOPS improves macro Avg@8 over FedAvg by 4.56--14.57 percentage points across the evaluated student models and local learning rates.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-29 in cs.DC).
Categorie arXiv: Machine Learning (cs.LG) ; Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2609.32573 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T21:53:13.748Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).