Metraipolis › archive › dc_5aee33dea254
Do Multilingual Encoders Produce Language-Consistent Semantic IDs? — arXiv:2610.01139
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.01139
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T13:57:11.433Z
- SHA-256 of the archived text
423c51ba7c6080df77f6a09062d3d378311cc99f710a041fcbbccd18c44c39b9- Stable URL of this page
https://metraipolis.com/documento/dc_5aee33dea254
How to cite
Do Multilingual Encoders Produce Language-Consistent Semantic IDs? — arXiv:2610.01139 — arXiv (en), https://arxiv.org/abs/2610.01139 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5aee33dea254 (Metraipolis archive dc_5aee33dea254, sha256 423c51ba7c6080df77f6a09062d3d378311cc99f710a041fcbbccd18c44c39b9)
BibTeX
@misc{dc5aee33dea254,
title = {Do Multilingual Encoders Produce Language-Consistent Semantic IDs? — arXiv:2610.01139},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.01139},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5aee33dea254; sha256 423c51ba7c6080df77f6a09062d3d378311cc99f710a041fcbbccd18c44c39b9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01139. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Semantic IDs (SIDs) compress item embeddings into discrete code sequences used in generative retrieval. We ask whether a multilingual encoder is sufficient for different-language renderings of the same product to receive language-consistent SIDs. Using Amazon ESCI listings rendered in English, Spanish, and Japanese, we test whether translations remain close to their English source, whether residual quantization is unusually sensitive to translation-induced movement, and whether multilingual or language-balanced quantizer fitting improves SID agreement. Multilingual E5 places translations measurably apart: under an English-heavy fit, a Japanese translation preserves the first SID code of its English counterpart in only 7.7% of cases, compared with 89.0% for an English rewording. Distance-matched product-directed controls produce nearly the same full-SID mismatch as translation, providing no evidence that the quantizer selectively amplifies language directions. Balancing the fitting mixture makes codebook use more uniform but further reduces cross-lingual prefix agreement: Spanish first-code consistency falls from 28.3% to 6.6%, while an English-only fit preserves it for 67.6% of Spanish translations. These results show that multilingual exposure and balanced codebook use alone do not guarantee language-consistent SIDs. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CL). Categorie arXiv: Information Retrieval (cs.IR) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.01139 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T12:15:34.822Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).