Metraipolis › archive › dc_86c047de807d

Cross-Lingual Alignment for Decoder-Only Models using MoE Routers — arXiv:2610.01921

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.01921
License (as archived)
CC0-1.0
Archived
2026-10-06T12:15:34.306Z
SHA-256 of the archived text
a70d272a683750b10e43456fec36e357c51d8398d6b9840234ac8d41f47f1e4f
Stable URL of this page
https://metraipolis.com/documento/dc_86c047de807d

How to cite

Cross-Lingual Alignment for Decoder-Only Models using MoE Routers — arXiv:2610.01921 — arXiv (en), https://arxiv.org/abs/2610.01921 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_86c047de807d (Metraipolis archive dc_86c047de807d, sha256 a70d272a683750b10e43456fec36e357c51d8398d6b9840234ac8d41f47f1e4f)
BibTeX
@misc{dc86c047de807d,
  title = {Cross-Lingual Alignment for Decoder-Only Models using MoE Routers — arXiv:2610.01921},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01921},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_86c047de807d; sha256 a70d272a683750b10e43456fec36e357c51d8398d6b9840234ac8d41f47f1e4f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01921. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Cross-lingual contrastive learning has been a core component of multilingual encoder training, but the ability to explicitly align representations is not possible in decoder-only LLMs because of varying multilingual tokenization. However, a growing amount of research suggests that even in LLMs, higher cross-lingual representational alignment leads to improved cross-lingual transfer. In this paper, we propose a novel approach to reimagine cross-lingual contrastive learning given the architectural constraints of modern LLMs. Rather than applying an auxiliary alignment loss on hidden states, we propose using the outputs of the mixture-of-experts (MoE) routers as the target for alignment. Router outputs lend themselves better to pooling over many tokens, enabling more reliable cross-lingual comparisons at the sequence-level. Controlled continual pre-training experiments on four open-source MoEs show that incorporating this routing loss also aligns the underlying hidden representations across languages. Most importantly, this loss improves multilingual performance on our diverse evaluation suite, demonstrating the potential of cross-lingual MoE router alignment.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.01921 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.229Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).