Metraipolis › archive › dc_218190843c00

Routing in Gradient Space: Balanced Usage Is Not Expert Specialization — arXiv:2609.36724

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.36724
License (as archived): CC0-1.0
Archived: 2026-10-07T17:29:25.162Z
SHA-256 of the archived text: e5af54cb15fbc5186c6b485b07d45dbf0f0038cec85cbcb7592628692db24882
Stable URL of this page: https://metraipolis.com/documento/dc_218190843c00

How to cite

Routing in Gradient Space: Balanced Usage Is Not Expert Specialization — arXiv:2609.36724 — arXiv (en), https://arxiv.org/abs/2609.36724 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_218190843c00 (Metraipolis archive dc_218190843c00, sha256 e5af54cb15fbc5186c6b485b07d45dbf0f0038cec85cbcb7592628692db24882)

BibTeX

@misc{dc218190843c00,
  title = {Routing in Gradient Space: Balanced Usage Is Not Expert Specialization — arXiv:2609.36724},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36724},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_218190843c00; sha256 e5af54cb15fbc5186c6b485b07d45dbf0f0038cec85cbcb7592628692db24882},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36724. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Sparse expert models can distribute traffic evenly while still grouping incompatible training signals within the same experts. We study routing as a gradient-partitioning problem and introduce gradient-aligned routing (GAR), whose load-normalized router objective rewards grouping observations with aligned gradients. On five multi-task text-classification mixtures, we compare GAR with task-loss-only routing, gradient-combination and gradient-conflict methods, and load-balancing losses. With a fully trainable RoBERTa backbone and classification-head experts, GAR has the highest aggregate validation accuracy, 1.07 percentage points above task-loss-only routing. With frozen DeBERTa and Qwen3-1.7B backbones and low-rank adapter experts, it again ranks first, 1.10 points above task-loss-only routing, with better-balanced expert load and higher gradient-mass purity, the share of each expert's gradient-norm mass from its dominant task; the load-balancing losses flatten load further but leave this purity near its task-loss-only level. Top-1 routing, trainable full-parameter feed-forward network (FFN) experts, and a larger backbone also show positive aggregate gains. The results distinguish expert-load balance from gradient-based routing organization and indicate the predictive value of gradient-informed routing in multi-task text classification.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.36724 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:59:48.226Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).