Metraipolis › archive › dc_ff0a708addec

OSCAR: Order-aware Scoring and Calibration for AI Rankings — arXiv:2609.24128

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.24128
License (as archived): CC0-1.0
Archived: 2026-10-09T07:08:44.045Z
SHA-256 of the archived text: ac1485984c5212474c4881fb160412b3cbfd95d36445de95dcd58154c837cc32
Stable URL of this page: https://metraipolis.com/documento/dc_ff0a708addec

How to cite

OSCAR: Order-aware Scoring and Calibration for AI Rankings — arXiv:2609.24128 — arXiv (en), https://arxiv.org/abs/2609.24128 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ff0a708addec (Metraipolis archive dc_ff0a708addec, sha256 ac1485984c5212474c4881fb160412b3cbfd95d36445de95dcd58154c837cc32)

BibTeX

@misc{dcff0a708addec,
  title = {OSCAR: Order-aware Scoring and Calibration for AI Rankings — arXiv:2609.24128},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.24128},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ff0a708addec; sha256 ac1485984c5212474c4881fb160412b3cbfd95d36445de95dcd58154c837cc32},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24128. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Judge-specific sensitivity is useful for aggregating pairwise LLM evaluations, but its interpretation depends on which systematic presentation effects the ranking model includes. We introduce OSCAR, an order-aware framework for scoring and calibrating AI rankings, and study position as one such effect. In released judgments from 18 evaluators, the all-response A-minus-B score difference ranges from $-63.11$ to $98.31$ percentage points. Matching question text, response texts, candidate identities, and judge within the released table gives an overall difference of $24.22$ points (95% interval $[22.90,25.54]$), conditional on the released text mapping. A controlled calculation isolates the potential consequence: with true sensitivity fixed at one, omitting a position intercept of four reduces the population-optimal slope to $0.0771$. We extend sensitivity-based ranking with judge-specific position, length, and family terms, characterize local omission-induced displacement and an identification failure, and propagate prompt-cluster uncertainty to adjusted comparisons. Across four released datasets, position provides the largest stand-alone predictive improvement. Refitting bootstrap comparisons show more selective gains from the full model over position-only adjustment. In dependent binary simulations, adjusting both the mean and covariance yields 94.4--95.2% coverage; correcting either alone is insufficient. At $N=10{,}000$, OSCAR reduces mean neutral-target RMSE from $0.1158$ under the sensitivity-only model to $0.0237$.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.LG).
Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG); Applications (stat.AP).
Pagina del lavoro: https://arxiv.org/abs/2609.24128 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:58:09.629Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).