Metraipolis › archive › dc_0ee23fedcfd7

Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format — arXiv:2609.09882

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.09882
License (as archived): CC0-1.0
Archived: 2026-10-16T07:24:34.316Z
SHA-256 of the archived text: 4d4903cef0541c9ef90d0bb42a575e8cd01447820cf4ebd0fa8c780083ce5e39
Stable URL of this page: https://metraipolis.com/documento/dc_0ee23fedcfd7

How to cite

Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format — arXiv:2609.09882 — arXiv (en), https://arxiv.org/abs/2609.09882 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0ee23fedcfd7 (Metraipolis archive dc_0ee23fedcfd7, sha256 4d4903cef0541c9ef90d0bb42a575e8cd01447820cf4ebd0fa8c780083ce5e39)

BibTeX

@misc{dc0ee23fedcfd7,
  title = {Scored vs. Generated Readouts in Behavioral Language Models: An Empirical Study of Elicitation Format — arXiv:2609.09882},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.09882},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0ee23fedcfd7; sha256 4d4903cef0541c9ef90d0bb42a575e8cd01447820cf4ebd0fa8c780083ce5e39},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.09882. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language models fine-tuned on customer behavior can predict outcomes and generate explanations, but these readouts are often treated as interchangeable. Holding model checkpoint and prompt content fixed, we compare probabilities obtained by scoring answer tokens with predictions generated after a written rationale. Across 13 model-domain cells covering four retail tasks in three markets, including two using fully public data and checkpoints, the scored readout ranks outcomes more accurately in 12 of 13 cells (two-sided sign test, p approximately 0.003), by 1.5 to 14.5 points in area under the receiver operating characteristic curve (AUC). Paired bootstrap confidence intervals exclude zero in every newly measured cell. The gap varies with task-specific supervision and mismatch between training and serving formats, ranging from -2.2 points for an untuned base model to +13.7 for rationale-format supervision. Analysis of approximately 9,000 rationales identifies two correlates: reduced reliance on the dominant predictive feature and convergence on stock formulations. Probability saturation does not track the gap. A third readout, eliciting a probability before any verdict, improves calibration (Brier score from 0.47 to 0.15) while ranking within noise of scoring, but only for outcome rates represented in training; it is worse than scoring when the scored head is already calibrated. We interpret these differences through the objectives matched by each readout, identify training choices that narrow the gap, and propose retaining generated rationales while sourcing ranking from the scored head.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-10 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.09882 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T07:00:16.225Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).