Metraipolis › archive › dc_015b216a70fb
Twenty Days Later: A Pre-Registered Test of History-Based Forecasts of Brand Mentions in Gemini Answers — arXiv:2610.19608
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.19608
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T20:39:04.197Z
- SHA-256 of the archived text
8c210b389f8449a9eb2550e78d2d8406159ca574dedafd94f9ea5b32ad5e7668- Stable URL of this page
https://metraipolis.com/documento/dc_015b216a70fb
How to cite
Twenty Days Later: A Pre-Registered Test of History-Based Forecasts of Brand Mentions in Gemini Answers — arXiv:2610.19608 — arXiv (en), https://arxiv.org/abs/2610.19608 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_015b216a70fb (Metraipolis archive dc_015b216a70fb, sha256 8c210b389f8449a9eb2550e78d2d8406159ca574dedafd94f9ea5b32ad5e7668)
BibTeX
@misc{dc015b216a70fb,
title = {Twenty Days Later: A Pre-Registered Test of History-Based Forecasts of Brand Mentions in Gemini Answers — arXiv:2610.19608},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.19608},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_015b216a70fb; sha256 8c210b389f8449a9eb2550e78d2d8406159ca574dedafd94f9ea5b32ad5e7668},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19608. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Counts of which brands a large language model names are useful only if they hold long enough to act on. We test this with a pre-registered replication. The same 162 requests (54 questions in English, Estonian and Finnish about three software categories, four of twelve candidate brands per question, three repeats) were sent to gemini-3.1-pro-preview in three waves within about ten hours on 22 and 23 September 2026, and again on 13 October 2026, 20.3 days after the third wave, with identical request bodies. The plan and its decision rule were frozen, hashed and registered on OSF before the first call. The outcome is a lexical mention label per answer and brand, 648 per wave. The primary forecast, each brand-and-question cell's mention rate over the nine earlier answers, scored Brier 0.0281 and log loss 0.126, against Brier 0.0259 for the six-answer forecast of the 8-hour wave and a registered ceiling of 0.0559. It beat brand history (0.104), the overall rate (0.234) and a constant 0.5 (0.250) on both scores and in all nine question families (exact sign-flip p = 0.0039), so durability holds under the registered rule. A forecast frozen from the first wave alone scored 0.0599, 0.0587 and 0.0604 at horizons of 1.5 hours, 9.6 hours and 20.7 days. Of the 202 cells unanimous in the 8-hour wave, 194 were unchanged at D1 and none reversed. The latest wave alone was nearly as accurate as the pooled history (Brier 0.0292; pooled lower in 5 of 9 families, p = 0.28). Every answer reported the requested model version, and none issued a search query although search was enabled. A blind human check of 96 labels agreed with the rule on 94; both disagreements were adjudicated as rater omissions, so after adjudication the rule matched on 96 of 96. The test covers one model, one interval and twelve known brands, and estimates no decay rate. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Information Retrieval (cs.IR) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.19608 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T19:36:18.294Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).