Metraipolis › archive › dc_98022ea0735d

Evaluating Communicative Success in Machine-Translated Conversation — arXiv:2609.19885

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.19885
License (as archived)
CC0-1.0
Archived
2026-10-10T03:48:07.865Z
SHA-256 of the archived text
eaaaacd7224949343110fa61511d557e99e19a69472400aca0aff4060b4e260e
Stable URL of this page
https://metraipolis.com/documento/dc_98022ea0735d

How to cite

Evaluating Communicative Success in Machine-Translated Conversation — arXiv:2609.19885 — arXiv (en), https://arxiv.org/abs/2609.19885 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_98022ea0735d (Metraipolis archive dc_98022ea0735d, sha256 eaaaacd7224949343110fa61511d557e99e19a69472400aca0aff4060b4e260e)
BibTeX
@misc{dc98022ea0735d,
  title = {Evaluating Communicative Success in Machine-Translated Conversation — arXiv:2609.19885},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.19885},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_98022ea0735d; sha256 eaaaacd7224949343110fa61511d557e99e19a69472400aca0aff4060b4e260e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.19885. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Interpreter agents built on machine translation (MT) increasingly mediate live conversation between people who do not share a language, yet we still evaluate them with metrics built for isolated sentences, which measure fidelity rather than whether communication succeeds. We introduce a reusable three-layer checklist-and-judge framework that evaluates interpreter-mediated conversation across semantic, pragmatic, and cultural-social dimensions, covering the naturalness, intent, and social appropriateness that fidelity metrics leave unmeasured. It runs in both single-turn and interactive multi-turn settings, where simulated users reply to translated messages as the conversation unfolds and each turn is scored alongside the conversation as a whole. We extensively validate it through controlled perturbations, cross-judge comparisons, and human annotations. Our main single-turn benchmark evaluates 10 interpreter setups across Arabic, Bengali, Indonesian, and Korean from 5,624 OpenSubtitles-derived scenarios spanning 12 translation directions, and our multi-turn study covers all 6 language pairs in scripted and live modes. Results show a consistent decline from semantic to pragmatic and cultural-social success, while conventional MT metrics overlook failures among stronger interpreters, and prompt ablations show that scenario context, structured instructions, and cultural context improve communicative success, although gains vary across setups. Our work thus provides an evaluation framework and benchmark for interpreter agents in conversation, and highlights the importance of communicative success alongside existing translation metrics.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.19885 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T03:24:08.141Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).