Metraipolis › archive › dc_cc7bdaa4ed82
The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations — arXiv:2609.04047
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.04047
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T01:44:40.902Z
- SHA-256 of the archived text
7b27c627f9c355890f21e3ba1a4367959d90db99cc7f64004575d3c392551a25- Stable URL of this page
https://metraipolis.com/documento/dc_cc7bdaa4ed82
How to cite
The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations — arXiv:2609.04047 — arXiv (en), https://arxiv.org/abs/2609.04047 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cc7bdaa4ed82 (Metraipolis archive dc_cc7bdaa4ed82, sha256 7b27c627f9c355890f21e3ba1a4367959d90db99cc7f64004575d3c392551a25)
BibTeX
@misc{dccc7bdaa4ed82,
title = {The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations — arXiv:2609.04047},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.04047},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cc7bdaa4ed82; sha256 7b27c627f9c355890f21e3ba1a4367959d90db99cc7f64004575d3c392551a25},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04047. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Background: Researchers increasingly use repeated identical prompts to audit stochastic variation in large language model (LLM) brand recommendations, yet no standardized protocol exists for setting iteration counts, selecting stability metrics, or establishing reliability thresholds. Objective: We formalize the Dice Roll Method as a reusable protocol for repeated-query auditing of LLM brand recommendations, grounded in a generative model of temperature-scaled nucleus sampling. Methods: Total response variance is decomposed into sampling, prompt-phrasing, run-to-run, and model-version components. The stack: a negative-binomial mixed model with iterations as repeated measures; Cliff's delta as the distribution-free effect size; dependence-preserving bootstrap; simulation-based power; a generalizability-theory decomposition; drift diagnostics on pinned snapshots. We reanalyse five brand-recommendation auditing studies: approximately 190,000 observations, 270+ brands, 6 languages, iteration counts 5 to 40. Results: The recomputed D-study supports no fixed tiers. Across 250 distinct questions (S5), G is 0.96 at five runs and 1 to 2 runs reach 0.80, also within each model. For five versions of one question (S1 Valentine's subset, Gemini) it is 0.27 (95% CI 0.00 to 0.52), which fixes no run count. Runs follow from a pilot: n = 4 x residual / cell variance. Simulated single-cell power for a large Cliff's delta is 0.15 to 0.19 at five runs and 0.8 at 15 to 25. The four metric families (count, set, embedding, PASOR) are complementary. A pre-registered external validation on three independent corpora reproduces the D-study prediction in 37 of 39 cells with no failures; fixed tiers do not transfer. Conclusion: The protocol gives repeated-query auditing of LLM brand recommendations a statistically principled footing under the conditional, non-Gaussian structure of real autoregressive generation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.CL). Categorie arXiv: Information Retrieval (cs.IR) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.04047 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T01:11:43.623Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).