Metraipolis › archive › dc_3fda84e3ed64

LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian — arXiv:2610.00406

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00406
License (as archived)
CC0-1.0
Archived
2026-10-06T13:53:55.766Z
SHA-256 of the archived text
2db55a903ace6e6ccbd93f66b94159796d402cc2bbb0a28fc32a65dcf2f1c2ad
Stable URL of this page
https://metraipolis.com/documento/dc_3fda84e3ed64

How to cite

LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian — arXiv:2610.00406 — arXiv (en), https://arxiv.org/abs/2610.00406 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3fda84e3ed64 (Metraipolis archive dc_3fda84e3ed64, sha256 2db55a903ace6e6ccbd93f66b94159796d402cc2bbb0a28fc32a65dcf2f1c2ad)
BibTeX
@misc{dc3fda84e3ed64,
  title = {LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian — arXiv:2610.00406},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00406},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3fda84e3ed64; sha256 2db55a903ace6e6ccbd93f66b94159796d402cc2bbb0a28fc32a65dcf2f1c2ad},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00406. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating Retrieval-Augmented Generation (RAG) systems remains a challenge for Low-Resource Languages (LRLs), where standard reference-based metrics fall short. This paper investigates the viability of the "LLM-as-a-Judge" paradigm for Romanian by adapting the Ragas framework using next-generation models (Gemini 2.5 and Gemini 3). We introduce AdminRo-Eval, a curated dataset of Romanian administrative documents annotated by native speakers, to serve as a ground truth for benchmarking automated evaluators. We compare three evaluation methodologies - direct scoring, comparative ranking, and granular decomposition - across metrics for Faithfulness, Answer Relevance, and Context Relevance. Our findings reveal that evaluation strategies must be metric-specific: granular decomposition achieves the highest human alignment for Faithfulness (96% with Gemini 2.5 Pro), while comparative ranking outperforms in Answer Relevance (90%). Furthermore, we demonstrate that while lightweight models struggle with complex reasoning in LRLs, the Gemini 2.5 Pro architecture establishes a robust, transferable baseline for automated Romanian RAG evaluation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.00406 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T12:15:34.618Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).