Metraipolis › archive › dc_aa69ed39352b

Can We Triage LLM Translation Errors in Classical Texts Without Human References? Source Novelty, GEMBA Scoring, and Budgeted Review through Pali-to-English Translation — arXiv:2609.14963

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.14963
License (as archived)
CC0-1.0
Archived
2026-10-15T23:15:37.805Z
SHA-256 of the archived text
0501f3e5a1de3c7eb6a2c9d50a072e57ef69b0be587935892c5b623d0da494bf
Stable URL of this page
https://metraipolis.com/documento/dc_aa69ed39352b

How to cite

Can We Triage LLM Translation Errors in Classical Texts Without Human References? Source Novelty, GEMBA Scoring, and Budgeted Review through Pali-to-English Translation — arXiv:2609.14963 — arXiv (en), https://arxiv.org/abs/2609.14963 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aa69ed39352b (Metraipolis archive dc_aa69ed39352b, sha256 0501f3e5a1de3c7eb6a2c9d50a072e57ef69b0be587935892c5b623d0da494bf)
BibTeX
@misc{dcaa69ed39352b,
  title = {Can We Triage LLM Translation Errors in Classical Texts Without Human References? Source Novelty, GEMBA Scoring, and Budgeted Review through Pali-to-English Translation — arXiv:2609.14963},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.14963},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aa69ed39352b; sha256 0501f3e5a1de3c7eb6a2c9d50a072e57ef69b0be587935892c5b623d0da494bf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.14963. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As large language models become capable translators of classical texts, a key challenge is deciding which outputs need expert review when no human reference exists. This study tests reference-free error triage through Pali-to-English translation. Three LLMs translated 15,493 passages. Five signals were compared: source novelty, source-candidate embedding distance, peer-translation disagreement, English-to-Pali backtranslation, and no-reference GEMBA scoring. Signals were calibrated on a 3,000-item reference-informed LLM-adjudicated sample and checked against a 500-item author-adjudicated anchor. Human references supported calibration and validation only; they were never used to compute the risk signals. Source novelty was a useful source-side risk prior but not a per-candidate error detector. Peer disagreement and backtranslation provided secondary signal. The strongest method was no-reference GEMBA scoring by a panel of models generally regarded as stronger than the translators: reviewing the top 10% by GEMBA risk captured 81.6% of panel-major errors in the calibration set. GEMBA also remained the best reference-free signal against the author anchor. A same-tier panel, with self-scoring excluded, remained useful but performed worse, indicating that evaluator strength matters beyond the prompt alone. A budgeted workflow is proposed, combining source novelty, peer disagreement, and stronger candidate-aware judging to allocate human review. Transfer to other classical languages, including Latin, Ancient Greek, and Sanskrit, remains to be tested.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.14963 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:51:09.915Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).