Metraipolis › archive › dc_01ebb6e00ddb

Templated or fully synthetic? Prompt construction as a confound in measuring LLM political stance beyond writing assistance — arXiv:2608.11008

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.11008
License (as archived)
CC0-1.0
Archived
2026-10-19T06:14:24.809Z
SHA-256 of the archived text
e52913abe5a4efdb6285663fd509fff44ed8bb84164be1f340aaf699e811d77e
Stable URL of this page
https://metraipolis.com/documento/dc_01ebb6e00ddb

How to cite

Templated or fully synthetic? Prompt construction as a confound in measuring LLM political stance beyond writing assistance — arXiv:2608.11008 — arXiv (en), https://arxiv.org/abs/2608.11008 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_01ebb6e00ddb (Metraipolis archive dc_01ebb6e00ddb, sha256 e52913abe5a4efdb6285663fd509fff44ed8bb84164be1f340aaf699e811d77e)
BibTeX
@misc{dc01ebb6e00ddb,
  title = {Templated or fully synthetic? Prompt construction as a confound in measuring LLM political stance beyond writing assistance — arXiv:2608.11008},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.11008},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_01ebb6e00ddb; sha256 e52913abe5a4efdb6285663fd509fff44ed8bb84164be1f340aaf699e811d77e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.11008. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Political stance detection in LLMs has long been dominated by closed-ended, multiple-choice political survey questions---originally designed for humans, and thus lacks the realism and nuance of human-AI interactions in the wild, while also being susceptible to sandbagging. The recent IssueBench framework substantially mitigates these limitations with templated prompts anchored in real-world chat logs. Given the rise in non-work-related use of GenAI assistants, we extend IssueBench beyond writing assistance to include two additional tasks, information seeking and opinion sharing. We argue that templated prompts still lack the nuance of real ones, especially for open-ended tasks, and remain recognisable as evaluation artefacts. We propose the use of fully synthetic (LLM-generated) prompts, produced under detailed instructions with real prompts as seeds. We assess the ecological validity of real, templated, and LLM-generated prompts in a small-scale study covering 3 highly contested policy issues and 3 recent geopolitical conflicts. Human and LLM annotators rank LLM-generated prompts as no less realistic than real ones and clearly more realistic than templated ones, and find that they carry their intended intent and stance more clearly; the LLMs separate templated prompts from the other two far more sharply than the humans do. In a case study, templated and LLM-generated prompts yield systematically different stance estimates for the same model, most visibly under neutral framings, where templated prompts overstate the model's leaning in the direction encoded by the topic-and-stance text (filler) slotted into their templates.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Computers and Society (cs.CY).
Pagina del lavoro: https://arxiv.org/abs/2608.11008 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T06:08:50.338Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).