Metraipolis › archive › dc_b858c288a653

How Proper Scoring Rules Shape LLM Forecasting — arXiv:2608.28482

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.28482
License (as archived)
CC0-1.0
Archived
2026-10-17T10:48:21.894Z
SHA-256 of the archived text
79b8fb915c194113425dcc0942cc010f45dbe2dfe68b523b9476f1be945e568e
Stable URL of this page
https://metraipolis.com/documento/dc_b858c288a653

How to cite

How Proper Scoring Rules Shape LLM Forecasting — arXiv:2608.28482 — arXiv (en), https://arxiv.org/abs/2608.28482 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b858c288a653 (Metraipolis archive dc_b858c288a653, sha256 79b8fb915c194113425dcc0942cc010f45dbe2dfe68b523b9476f1be945e568e)
BibTeX
@misc{dcb858c288a653,
  title = {How Proper Scoring Rules Shape LLM Forecasting — arXiv:2608.28482},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.28482},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b858c288a653; sha256 79b8fb915c194113425dcc0942cc010f45dbe2dfe68b523b9476f1be945e568e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28482. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

This paper evaluates how reward function choice shapes the performance and behavior of LLM forecasters. We compare five proper scoring rules as training objectives for binary forecasts of resolved real-world events. Although the rules share the same theoretical incentive for truthful probability reporting, the resulting models differ in calibration, probability use, and estimated profiles of bias, information, and noise, with smaller differences in aggregate accuracy and discrimination. The Brier-trained model has the lowest observed Brier score and highest AUC-ROC, while the log-trained model has the highest observed log score and lowest calibration error. Models with similar aggregate performance also reach that performance through different combinations of bias, information, and noise. Proper scoring rules therefore need not behave interchangeably as training objectives. Reward choice may shape not only how well an LLM forecasts, but how its forecasting errors are structured.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.28482 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:33:42.537Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).