Metraipolis › archive › dc_5aee33dea254

Do Multilingual Encoders Produce Language-Consistent Semantic IDs? — arXiv:2610.01139

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.01139
License (as archived)
CC0-1.0
Archived
2026-10-06T13:57:11.433Z
SHA-256 of the archived text
423c51ba7c6080df77f6a09062d3d378311cc99f710a041fcbbccd18c44c39b9
Stable URL of this page
https://metraipolis.com/documento/dc_5aee33dea254

How to cite

Do Multilingual Encoders Produce Language-Consistent Semantic IDs? — arXiv:2610.01139 — arXiv (en), https://arxiv.org/abs/2610.01139 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5aee33dea254 (Metraipolis archive dc_5aee33dea254, sha256 423c51ba7c6080df77f6a09062d3d378311cc99f710a041fcbbccd18c44c39b9)
BibTeX
@misc{dc5aee33dea254,
  title = {Do Multilingual Encoders Produce Language-Consistent Semantic IDs? — arXiv:2610.01139},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01139},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5aee33dea254; sha256 423c51ba7c6080df77f6a09062d3d378311cc99f710a041fcbbccd18c44c39b9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01139. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Semantic IDs (SIDs) compress item embeddings into discrete code sequences used in generative retrieval. We ask whether a multilingual encoder is sufficient for different-language renderings of the same product to receive language-consistent SIDs. Using Amazon ESCI listings rendered in English, Spanish, and Japanese, we test whether translations remain close to their English source, whether residual quantization is unusually sensitive to translation-induced movement, and whether multilingual or language-balanced quantizer fitting improves SID agreement. Multilingual E5 places translations measurably apart: under an English-heavy fit, a Japanese translation preserves the first SID code of its English counterpart in only 7.7% of cases, compared with 89.0% for an English rewording. Distance-matched product-directed controls produce nearly the same full-SID mismatch as translation, providing no evidence that the quantizer selectively amplifies language directions. Balancing the fitting mixture makes codebook use more uniform but further reduces cross-lingual prefix agreement: Spanish first-code consistency falls from 28.3% to 6.6%, while an English-only fit preserves it for 67.6% of Spanish translations. These results show that multilingual exposure and balanced codebook use alone do not guarantee language-consistent SIDs.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CL).
Categorie arXiv: Information Retrieval (cs.IR) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.01139 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T12:15:34.822Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).