Metraipolis › archive › dc_400cd6d93af4

QUORUM: QUality-Optimized Routing Using Multiple annotators — arXiv:2608.27974

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27974
License (as archived)
CC0-1.0
Archived
2026-10-17T10:56:34.824Z
SHA-256 of the archived text
ffe887fc2e472c2ecd5ec1a8209cd653d217c679813b0fcdb27bf6b431e4b1b7
Stable URL of this page
https://metraipolis.com/documento/dc_400cd6d93af4

How to cite

QUORUM: QUality-Optimized Routing Using Multiple annotators — arXiv:2608.27974 — arXiv (en), https://arxiv.org/abs/2608.27974 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_400cd6d93af4 (Metraipolis archive dc_400cd6d93af4, sha256 ffe887fc2e472c2ecd5ec1a8209cd653d217c679813b0fcdb27bf6b431e4b1b7)
BibTeX
@misc{dc400cd6d93af4,
  title = {QUORUM: QUality-Optimized Routing Using Multiple annotators — arXiv:2608.27974},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27974},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_400cd6d93af4; sha256 ffe887fc2e472c2ecd5ec1a8209cd653d217c679813b0fcdb27bf6b431e4b1b7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27974. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Data annotation remains a central bottleneck in natural language processing, requiring human effort to obtain high-quality labels at scale. While Large Language Models (LLMs) offer a fast and cost-effective alternative, their reliability is highly instance-dependent: they perform well on simple inputs but often fail on examples requiring nuanced reasoning or contextual understanding. In this work, we address this challenge with QUORUM (QUality-Optimized Routing Using Multiple annotators), a budget-aware routing framework that dynamically assigns each instance to human or LLM annotators under a fixed annotation budget. Unlike prior approaches relying on model confidence or uncertainty estimates, QUORUM leverages feature-based signals to estimate instance difficulty and supports multiple annotations per instance, combining them through agreement-based rewards to improve reliability. We evaluate QUORUM across diverse closed- and open-ended annotation tasks in English and multilingual settings, and QUORUM improves annotation quality by up to 34.4% while reducing costs by 8.8% over competing methods. Code can be found at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.27974 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:47:33.121Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).