Metraipolis › archive › dc_f8ccf3347caa
JABER: Sequential Quality Control for Human-AI Data Annotation under Shared Bias — arXiv:2611.02892
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02892
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T10:24:03.613Z
- SHA-256 of the archived text
ca0ea6e48d53efa1e4bc8707316a0d47194024c586d05dd8160828248a7205de- Stable URL of this page
https://metraipolis.com/documento/dc_f8ccf3347caa
How to cite
JABER: Sequential Quality Control for Human-AI Data Annotation under Shared Bias — arXiv:2611.02892 — arXiv (en), https://arxiv.org/abs/2611.02892 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f8ccf3347caa (Metraipolis archive dc_f8ccf3347caa, sha256 ca0ea6e48d53efa1e4bc8707316a0d47194024c586d05dd8160828248a7205de)
BibTeX
@misc{dcf8ccf3347caa,
title = {JABER: Sequential Quality Control for Human-AI Data Annotation under Shared Bias — arXiv:2611.02892},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02892},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f8ccf3347caa; sha256 ca0ea6e48d53efa1e4bc8707316a0d47194024c586d05dd8160828248a7205de},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02892. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
High-quality annotations are essential for training and evaluating artificial intelligence (AI) systems. Quality control (QC) often acquires additional human or AI judgments before accepting a label. Nevertheless, shared biases can cause correlated judgments, leading conventional methods to mistake repeated agreement for independent evidence of correctness. To address this challenge, we propose JABER (Joint Alias-Preserving Belief for Evidence Routing), a bias-aware sequential QC framework that maintains a joint Bayesian posterior over label correctness and shared biases to guide evidence acquisition and termination decisions. Building on this posterior, we develop a finite-horizon algorithm that selects evidence-acquisition and termination actions to minimize expected QC loss under resource constraints. Our theoretical analysis shows that conditional-independence updating can become arbitrarily confident in an incorrect label under persistent shared bias. We further identify when repeated inspections cannot distinguish a correct label from an incorrect label supported by shared bias, when obtaining a new label can resolve this ambiguity, and why modeling the dependence between label correctness and shared bias can be necessary for QC decisions. Our experiments show that JABER reduces expected QC loss by an average of 4.65% compared with a method maintaining separate label correctness and bias probabilities when shared bias makes repeated inspections unreliable. Further, JABER's finite-horizon planning, which evaluates outcomes of successive QC actions, reduces expected QC loss by an average of 10.05% compared with a one-step lookahead method. In real-API experiments on FEVER, JABER achieves zero labeling errors while reducing QC costs by 2.67% and 16.99% compared with marginal-based belief modeling and one-step lookahead methods, respectively. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.02892 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:19:49.919Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).