Metraipolis › archive › dc_2f24624c66bb
BITEM at the NTCIR-19 R2C2 Task: Predicting Confidence from Agentic RAG Pipeline Signals — arXiv:2609.37993
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.37993
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:27:58.885Z
- SHA-256 of the archived text
cca9ceb19a72b0c22c8adc4e86fe4e6926eac3c92a45e789c97d83a10c03b0c0- Stable URL of this page
https://metraipolis.com/documento/dc_2f24624c66bb
How to cite
BITEM at the NTCIR-19 R2C2 Task: Predicting Confidence from Agentic RAG Pipeline Signals — arXiv:2609.37993 — arXiv (en), https://arxiv.org/abs/2609.37993 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2f24624c66bb (Metraipolis archive dc_2f24624c66bb, sha256 cca9ceb19a72b0c22c8adc4e86fe4e6926eac3c92a45e789c97d83a10c03b0c0)
BibTeX
@misc{dc2f24624c66bb,
title = {BITEM at the NTCIR-19 R2C2 Task: Predicting Confidence from Agentic RAG Pipeline Signals — arXiv:2609.37993},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.37993},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2f24624c66bb; sha256 cca9ceb19a72b0c22c8adc4e86fe4e6926eac3c92a45e789c97d83a10c03b0c0},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37993. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The BITEM team entered both subtasks of the NTCIR-19 R2C2 task with a single agentic pipeline, in which a model searches, reads and records evidence over a movie corpus while an orchestrator holds the record and rules on what may be submitted. A claim is admitted only once an entailment cascade has checked it against the passage it cites, and an answer is released only once enough checked evidence stands behind it. Each question is run three or four times, every pass retrieving from a corpus stripped of what the earlier passes have already seen. The confidence filed with each answer is computed by the orchestrator from what the run leaves behind and is never asked of the model, which is offered no way to rate itself. The two retrieval runs placed 4th and 5th of 22, pooling the passes was worth 0.0709 nDCG@20, and the gain was largest on the multi-hop and post-processing-heavy questions, where the organisers rank the pooled run top of the field. Sixteen of the 25 answer runs were built on passages these two runs supplied, 12 of them filed by other teams. HMR rewards a system whose confidence is high where it answers right and low where it answers wrong. The pipeline reached an accuracy of 0.9219, 6th of 25, while the confidence filed with those answers gave an HMR of 0.4915, 13th. A few rules crafted over those same recorded signals, with no further model call and no further retrieval, raise that to an accuracy of 0.9375, 5th, and an HMR of 0.6985, 9th. Ranking on HMR alone can reward a system for answering wrongly with low confidence, so we propose accHMR, the accuracy multiplied by HMR, which reports the reward in proportion to the accuracy, and on which the revised rules would have scored 0.6549, 5th. For future work, fitting a model on the numbers the pipeline already produces, rather than writing such rules by hand, would be a real step forward. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Information Retrieval (cs.IR). Pagina del lavoro: https://arxiv.org/abs/2609.37993 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T15:54:54.057Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).