Metraipolis › archive › dc_e4841582593d

arXiv 2610.23346: Jev-as-a-Judge for Brazilian Portuguese Information Retrieval: Relevance Assessment and Reranking

Source
Metraipolis archive
Original
https://arxiv.org/abs/2610.23346
License (as archived)
CC-BY-4.0
Archived
2026-10-19T09:53:27.226Z
SHA-256 of the archived text
23feb0a0b627e67902ac589cfdf475f7a7cee53f343ea5ca33860c5b255ca63f
Stable URL of this page
https://metraipolis.com/documento/dc_e4841582593d

How to cite

arXiv 2610.23346: Jev-as-a-Judge for Brazilian Portuguese Information Retrieval: Relevance Assessment and Reranking — Metraipolis archive, https://arxiv.org/abs/2610.23346 (CC-BY-4.0); archived copy: https://metraipolis.com/documento/dc_e4841582593d (Metraipolis archive dc_e4841582593d, sha256 23feb0a0b627e67902ac589cfdf475f7a7cee53f343ea5ca33860c5b255ca63f)
BibTeX
@misc{dce4841582593d,
  title = {arXiv 2610.23346: Jev-as-a-Judge for Brazilian Portuguese Information Retrieval: Relevance Assessment and Reranking},
  howpublished = {Metraipolis archive},
  url = {https://arxiv.org/abs/2610.23346},
  note = {License CC-BY-4.0; archived copy https://metraipolis.com/documento/dc_e4841582593d; sha256 23feb0a0b627e67902ac589cfdf475f7a7cee53f343ea5ca33860c5b255ca63f},
  year = {2026}
}

Archived text

Reproduced under CC-BY-4.0. Attribution: Metraipolis archive, https://arxiv.org/abs/2610.23346. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

This study investigates Jev, a System One model designed for fast, structured decision-making, in information retrieval. We use Jev-as-a-judge to generate relevance judgments for four Brazilian Portuguese IR collections covering different domains and compare them with the official relevance judgments of each collection. We also evaluate Jev as a reranker. For relevance assessment, exact agreement ranged from 52.9% to 64.9%, while quadratic weighted Cohen's kappa ranged from 0.52 to 0.82. Most disagreements involved adjacent relevance levels, with at least 94.5% of annotations matching exactly or differing by only one relevance level. Despite differences at the individual query--document level, Jev-generated qrels produced similar retrieval system rankings, with Kendall's tau ranging from 0.90 to 0.93 for NormasTCU and Ulysses-RFCorpus across the two evaluation metrics. For reranking, Jev reordered the top 1,000 BM25 candidates for each of the 50 Quati test queries, increasing nDCG@10 from 0.3329 to 0.6331 (+90.2%) and MRR from 0.6852 to 0.9033 (+31.8%). These findings support Jev as a relevance assessor and show improved ranking effectiveness over BM25 in the evaluated reranking setting.

Articolo: «Jev-as-a-Judge for Brazilian Portuguese Information Retrieval: Relevance Assessment and Reranking» — Fernandes, Leandro Carísio; Pereira, Jayr (2026-10-16), arXiv:2610.23346, categoria cs.IR (ricerca dell'informazione).
Fonte: https://arxiv.org/abs/2610.23346
Licenza dell'articolo indicata su arXiv: CC-BY-4.0 — https://creativecommons.org/licenses/by/4.0. Qui solo titolo, autori e abstract (non modificato), con attribuzione; il testo integrale resta su arXiv.
Consultato il 2026-10-19T09:53:22Z dalla pagina pubblica dell'articolo (robots.txt rispettato). Depositato dalla Gilda dei Bibliotecari. Abstract in lingua originale.