Metraipolis › archive › dc_191af33619e2

Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026 — arXiv:2611.03525

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.03525
License (as archived)
CC0-1.0
Archived
2026-11-03T10:44:24.917Z
SHA-256 of the archived text
228318489e173c67e3dec9bea59a85eb400ceac3f5ca58f0f677325fbeb179cb
Stable URL of this page
https://metraipolis.com/documento/dc_191af33619e2

How to cite

Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026 — arXiv:2611.03525 — arXiv (en), https://arxiv.org/abs/2611.03525 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_191af33619e2 (Metraipolis archive dc_191af33619e2, sha256 228318489e173c67e3dec9bea59a85eb400ceac3f5ca58f0f677325fbeb179cb)
BibTeX
@misc{dc191af33619e2,
  title = {Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026 — arXiv:2611.03525},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03525},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_191af33619e2; sha256 228318489e173c67e3dec9bea59a85eb400ceac3f5ca58f0f677325fbeb179cb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03525. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Skill extraction benchmarks score models against a few human-annotated positives and treat every other skill in the vocabulary as equally wrong. At the scale of ESCO (13,939 skills) this binary view can reward models that rank the annotated positive high for the wrong reason, and hides semantically nonsensical predictions. The WorkRB Challenge at RecSys-HR 2026 replaces it with graded relevance: every candidate skill in a model's ranking window carries a 0 to 4 grade, and the leaderboard scores nDCG@100 against these grades. Since ternary judgements proved too coarse for objective annotation, relevance is decomposed into a cascade of four binary questions (same domain, activity mentioned, right granularity, replacement of a gold positive) that a smaller LLM judge can answer reliably; the labels are validated with blind human and frontier-model re-annotation. This paper documents the annotation methodology, the construction of five benchmark datasets (four with vacancy sentences as queries, one with lexically distant skill surface forms), the validation of the LLM cascade, and insights from the submissions. The field converged on multi-stage retrieve-and-rerank pipelines, whose gains come from clearing nonsense out of the ranking window rather than finding more positives: at the top of the leaderboard, the graded ordering is uncorrelated with what binary evaluation would have said. Graded labels expose much bigger gaps between top submissions and baselines than binary evaluation does, and show that semantic-understanding skill recommendation is still wide open. The clear next direction is bringing efficiency back: every submitted system we could size uses at least 42x, typically several hundred times, more parameters than the baselines it beats.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.03525 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:39:50.355Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).