Metraipolis › archive › dc_a4e328aa61d7
Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languages — arXiv:2608.24477
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.24477
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T06:18:06.999Z
- SHA-256 of the archived text
7f798cb61d20d4b94572d2c704897bbb90d79f70117addfb36d3b762e4c7d80e- Stable URL of this page
https://metraipolis.com/documento/dc_a4e328aa61d7
How to cite
Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languages — arXiv:2608.24477 — arXiv (en), https://arxiv.org/abs/2608.24477 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a4e328aa61d7 (Metraipolis archive dc_a4e328aa61d7, sha256 7f798cb61d20d4b94572d2c704897bbb90d79f70117addfb36d3b762e4c7d80e)
BibTeX
@misc{dca4e328aa61d7,
title = {Dataset Scarcity Limits Robust Evaluation of Multilingual Embedding Models: A Case Study of Slavic Languages — arXiv:2608.24477},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.24477},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a4e328aa61d7; sha256 7f798cb61d20d4b94572d2c704897bbb90d79f70117addfb36d3b762e4c7d80e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.24477. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Multilingual text embedding models enable cross-lingual transfer of knowledge across a wide range of NLP tasks, but their evaluation remains highly uneven across high-, mid- and low-resource languages. In this paper, we propose a two-dimensional framework, specifically tailored for analyzing multilingual embedding benchmarks under dataset scarcity, and apply it on the Slavic-language subset of the MTEB benchmark. The framework distinguishes between task-specific and cross-task evaluation, while jointly analyzing three complementary aspects: (1) ranking robustness, (2) model consistency, and (3) evidence strength. At the task-specific level, we evaluate the stability of model rankings under changes in ranking methodology and benchmark dataset composition. At the cross-task level, we assess the ability of models to generalize across diverse tasks within a language. To quantify the reliability of benchmark conclusions, we introduce an Evidence Strength Score that accounts for dataset availability, diversity, and robustness assessability. Our analysis reveals severe benchmark sparsity, with many Slavic language-task pairs relying on a single dataset or highly correlated benchmark collections, limiting the ability to draw robust conclusions. The cross-task analysis reveals a small group of highly transferable models, most notably llama-embed-nemotron-8b, multilingual-e5-large-instruct, and Qwen3-Embedding variants, that consistently perform well across Slavic languages and tasks. Overall, the results demonstrate that benchmark rankings and robustness conclusions must be interpreted jointly with certain notation of their evidence strength and highlight benchmark scarcity as a major obstacle to trustworthy multilingual evaluation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-26 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.24477 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T06:17:29.369Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).