Metraipolis › archive › dc_06efa033e55d

Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction — arXiv:2611.07735

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.07735
License (as archived): CC0-1.0
Archived: 2026-11-06T14:42:37.669Z
SHA-256 of the archived text: 325a77bc93239c91b0033505f96bb2a847aff176b284299149d6dce1f2d21588
Stable URL of this page: https://metraipolis.com/documento/dc_06efa033e55d

How to cite

Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction — arXiv:2611.07735 — arXiv (en), https://arxiv.org/abs/2611.07735 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_06efa033e55d (Metraipolis archive dc_06efa033e55d, sha256 325a77bc93239c91b0033505f96bb2a847aff176b284299149d6dce1f2d21588)

BibTeX

@misc{dc06efa033e55d,
  title = {Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction — arXiv:2611.07735},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07735},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_06efa033e55d; sha256 325a77bc93239c91b0033505f96bb2a847aff176b284299149d6dce1f2d21588},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07735. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As large language models (LLMs) are increasingly used to simulate human behavior, their operational validity is typically assessed through accuracy alone, implicitly treating human behavior as deterministic. We argue that on noisy digital trace data, a model's performance depends not only on predicting the correct outcome but on its calibration. We present a case study testing whether the token probabilities of open-weight LLMs can serve as confidence estimates in a Next Like Prediction task: predicting how a user of the Luxembourgish news platform this http URL rates a comment, conditioned on their prior interactions. Evaluating Ministral 3 (3B, 8B), we find that, with in-context learning only, the tested models are overconfident and are outperformed on accuracy and calibration by a simple statistical baseline. Accuracy and calibration do not move together: scaling from 3B to 8B improves accuracy while degrading calibration, and longer behavioral histories raise accuracy while confidence declines. Task-specific supervised fine-tuning acts as an effective calibration intervention, reducing Expected Calibration Error from 0.2 to 0.006 while improving accuracy from 0.536 to 0.758. We conclude that confidence estimation should be evaluated alongside accuracy to improve validity when using LLM-based agents.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.07735 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:18:42.687Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).