Metraipolis › archive › dc_3fda84e3ed64
LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian — arXiv:2610.00406
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.00406
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T13:53:55.766Z
- SHA-256 of the archived text
2db55a903ace6e6ccbd93f66b94159796d402cc2bbb0a28fc32a65dcf2f1c2ad- Stable URL of this page
https://metraipolis.com/documento/dc_3fda84e3ed64
How to cite
LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian — arXiv:2610.00406 — arXiv (en), https://arxiv.org/abs/2610.00406 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3fda84e3ed64 (Metraipolis archive dc_3fda84e3ed64, sha256 2db55a903ace6e6ccbd93f66b94159796d402cc2bbb0a28fc32a65dcf2f1c2ad)
BibTeX
@misc{dc3fda84e3ed64,
title = {LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian — arXiv:2610.00406},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.00406},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3fda84e3ed64; sha256 2db55a903ace6e6ccbd93f66b94159796d402cc2bbb0a28fc32a65dcf2f1c2ad},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00406. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Evaluating Retrieval-Augmented Generation (RAG) systems remains a challenge for Low-Resource Languages (LRLs), where standard reference-based metrics fall short. This paper investigates the viability of the "LLM-as-a-Judge" paradigm for Romanian by adapting the Ragas framework using next-generation models (Gemini 2.5 and Gemini 3). We introduce AdminRo-Eval, a curated dataset of Romanian administrative documents annotated by native speakers, to serve as a ground truth for benchmarking automated evaluators. We compare three evaluation methodologies - direct scoring, comparative ranking, and granular decomposition - across metrics for Faithfulness, Answer Relevance, and Context Relevance. Our findings reveal that evaluation strategies must be metric-specific: granular decomposition achieves the highest human alignment for Faithfulness (96% with Gemini 2.5 Pro), while comparative ranking outperforms in Answer Relevance (90%). Furthermore, we demonstrate that while lightweight models struggle with complex reasoning in LRLs, the Gemini 2.5 Pro architecture establishes a robust, transferable baseline for automated Romanian RAG evaluation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.00406 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T12:15:34.618Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).