Metraipolis › archive › dc_b145387ffd2a
When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation — arXiv:2609.08156
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.08156
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T12:48:40.330Z
- SHA-256 of the archived text
3215bc658cc97d7ac0c812e8096b124d215f733a66b9fc4befe9792aa7d1e5bf- Stable URL of this page
https://metraipolis.com/documento/dc_b145387ffd2a
How to cite
When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation — arXiv:2609.08156 — arXiv (en), https://arxiv.org/abs/2609.08156 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b145387ffd2a (Metraipolis archive dc_b145387ffd2a, sha256 3215bc658cc97d7ac0c812e8096b124d215f733a66b9fc4befe9792aa7d1e5bf)
BibTeX
@misc{dcb145387ffd2a,
title = {When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation — arXiv:2609.08156},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.08156},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b145387ffd2a; sha256 3215bc658cc97d7ac0c812e8096b124d215f733a66b9fc4befe9792aa7d1e5bf},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.08156. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Automatic translation quality metrics trained on general-domain corpora systematically fail on social media content, where communicative intent is encoded in culturally loaded expressions (internet slang, homophonic ciphers, and platform-specific idioms) rather than surface token patterns. We conduct a systematic empirical analysis demonstrating that standard metrics including COMET, XCOMET, and BERTScore exhibit near-zero or negative correlation with human cultural judgments, and even display a severity inversion in which scores increase as translation quality deteriorates. We further show that this failure extends to large language model judges: Qwen3-235B achieves Cohen's kappa of only 0.162, revealing that the bottleneck is not reasoning capacity but cultural grounding: models lack the domain-specific cultural knowledge needed to identify which aspects of a translation require scrutiny. To address this, we propose CuRIL, a reinforcement learning framework that internalizes cultural reasoning: cultural annotations are prepended inside the model's reasoning, excluded from policy gradients via a token-level loss mask, and injected with a probability that decays to zero over training, progressively forcing autonomous cultural judgment. On a 1,444-sample human-annotated social media translation benchmark, Qwen3-8B trained with CuRIL achieves Cohen's kappa 0.370 and Exact Match accuracy of 45.22%, approaching Gemini-3.1-Pro with 30x fewer parameters and surpassing models up to 235B in scale. We further demonstrate that our judge produces reliable reward signals for downstream translation optimization, reducing the low-quality translation rate by over 20 percentage points under independent human evaluation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.08156 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T12:30:25.163Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).