Metraipolis › archive › dc_edc8c366a47c
MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models — arXiv:2609.08663
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.08663
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T13:58:21.906Z
- SHA-256 of the archived text
0bb9c2c8d593ab1e66daadd2f2b9465a37e176890017e8002f5bf8f4cabd5c14- Stable URL of this page
https://metraipolis.com/documento/dc_edc8c366a47c
How to cite
MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models — arXiv:2609.08663 — arXiv (en), https://arxiv.org/abs/2609.08663 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_edc8c366a47c (Metraipolis archive dc_edc8c366a47c, sha256 0bb9c2c8d593ab1e66daadd2f2b9465a37e176890017e8002f5bf8f4cabd5c14)
BibTeX
@misc{dcedc8c366a47c,
title = {MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models — arXiv:2609.08663},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.08663},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_edc8c366a47c; sha256 0bb9c2c8d593ab1e66daadd2f2b9465a37e176890017e8002f5bf8f4cabd5c14},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.08663. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Universal multimodal embedding (UME) increasingly demands encoder's capacity for handling a broad range of tasks and modalities with increased complexity. Prior scaling methods either increase the representation size, retrieval effort, or scales the encoder into a heavy multimodal LLM. Recent works, such as Think-Then-Embed (TTE), explore scaling via reasoning tokens. However, embedding models are hard to scale up: increasing parameters directly tradeoffs for the large training batch size that contrastive learning needs, and retrieval has to be served under tight latency. Moreover, UME tasks are diverse in complexity, where scaling up embedders can bring significant redundant computation. In this work, we propose MOEMB, which instead scales UME along the expert axis through mixture-of-experts (MoE), growing encoder capacity while preserving single-vector, non-autoregressive encoding. Through a systematic study of the design space and training recipes for MoE-based UME, MoEMB sets a new state of the art on both MMEB-V2 and MRMR among models trained on public MMEB-family data: with only 3B active parameters, MoEMB surpasses TTE-based methods with >4x active parameters, using significantly less computes. To further improve the scalability and efficiency, we conduct the first comprehensive study of adaptive computation for MoE-based embedding, spanning diverse strategies across training-based and inference-only methods. Together, these results support expert scaling as an effective and efficient direction for UME, with adaptive computation further improving efficiency for MLLM-based embedding models towards large-scale retrieval and recommendation systems. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.08663 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T13:39:43.058Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).