Metraipolis › archive › dc_2cccf1564f7d
LLMAdBench: A Human Preference Benchmark for Advertising in LLM Responses — arXiv:2609.32533
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.32533
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:55:41.271Z
- SHA-256 of the archived text
315b3222bc3b1294ed38cf58a81a6062e28df45a69832f4a9543c093de3c8cf8- Stable URL of this page
https://metraipolis.com/documento/dc_2cccf1564f7d
How to cite
LLMAdBench: A Human Preference Benchmark for Advertising in LLM Responses — arXiv:2609.32533 — arXiv (en), https://arxiv.org/abs/2609.32533 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2cccf1564f7d (Metraipolis archive dc_2cccf1564f7d, sha256 315b3222bc3b1294ed38cf58a81a6062e28df45a69832f4a9543c093de3c8cf8)
BibTeX
@misc{dc2cccf1564f7d,
title = {LLMAdBench: A Human Preference Benchmark for Advertising in LLM Responses — arXiv:2609.32533},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.32533},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2cccf1564f7d; sha256 315b3222bc3b1294ed38cf58a81a6062e28df45a69832f4a9543c093de3c8cf8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32533. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Inserting advertisements (ads) into consumer-facing LLM output is emerging as a new business model, but there is little shared evidence on how such ad insertion should be evaluated or how it affects user preferences. We introduce LLMAdBench, a human-preference benchmark for studying advertising in LLM-generated content. The benchmark isolates a simple but practically important decision: given a user conversation, an LLM response, and a matched advertisement, where should the ad be placed? Our dataset compares pairs of responses that differ only in ad position while holding all other conditions fixed including the user query, base answer, advertisement, and disclosure condition. Human annotators evaluate each pair based on six criteria from both advertiser's and user's perspectives. The resulting benchmark contains more than 18000 human judgments across two disclosure conditions: explicitly labeling the ad as sponsored and merging it into the response without disclosure. We use LLMAdBench to evaluate eight frontier LLMs as preference judges and find that they are not reliable substitutes for human evaluation. Even the most stable models reverse roughly one quarter of their decisions when the presentation order is swapped, agreement across models is low, and their placement preferences differ systematically from those of human annotators. Moreover, LLMAdBench contains substantial learnable signal. In particular, a Qwen3-8B model fine-tuned on the human preferences improves substantially over its base model and outperforms all zero-shot frontier judges on the held-out prediction task. Beyond model evaluation, LLMAdBench provides quantitative evidence on the advertiser-user trade-off and shows that the sponsorship disclosure systematically changes users' preference over ad placement. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.32533 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:00.859Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).