Metraipolis › archive › dc_683d7eacd162
Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure — arXiv:2609.29445
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.29445
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T04:49:31.406Z
- SHA-256 of the archived text
222af26296582c3926f117e9ca85c1b19316900d357dafdeaee8d733006c9d61- Stable URL of this page
https://metraipolis.com/documento/dc_683d7eacd162
How to cite
Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure — arXiv:2609.29445 — arXiv (en), https://arxiv.org/abs/2609.29445 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_683d7eacd162 (Metraipolis archive dc_683d7eacd162, sha256 222af26296582c3926f117e9ca85c1b19316900d357dafdeaee8d733006c9d61)
BibTeX
@misc{dc683d7eacd162,
title = {Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure — arXiv:2609.29445},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.29445},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_683d7eacd162; sha256 222af26296582c3926f117e9ca85c1b19316900d357dafdeaee8d733006c9d61},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.29445. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We audit a multilingual affective generation benchmark eight instruction-tuned LLMs producing emoji summaries for 17,100 Bangla, English and Hindi sentences, with 6,960 human judgements and find its headline conclusions to be artefacts of the measurement instrument rather than properties of the systems. Treating annotators as a random rather than a fixed factor, no system differs significantly from any other ($F(7,14)=0.59$, $p=0.76$), although the conventional analysis declares 19 of 28 pairwise differences significant. Annotator identity explains far more rating variance than system identity, and the winning system changes whenever any single annotator is removed. The ordering that does emerge tracks output length: mean emoji count explains 78.7\% of between-system variance, and a within-item length-matched comparison over 2,599 pairs reverses the leaderboard. We further show that cross-provider anisotropy differences vanish under mean-centring, that per-language token costs change sign with the normalising unit, and that multi-view row-wise splits inflate macro-F1 by $3.1$ points and change the top-ranked system. In place of preference scoring we propose **emoji-affect decodability**, a reference-based probe whose rankings are stable to $\pm0.003$ macro-F1 across seeds. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-25 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.29445 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T04:27:20.062Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).