Metraipolis › archive › dc_3adf43aed06b

When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation — arXiv:2608.11843

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.11843
License (as archived)
CC0-1.0
Archived
2026-10-19T05:27:39.171Z
SHA-256 of the archived text
665118e819caaec5e67b2e409540ad4af2da53646f171dac100a039768c657dd
Stable URL of this page
https://metraipolis.com/documento/dc_3adf43aed06b

How to cite

When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation — arXiv:2608.11843 — arXiv (en), https://arxiv.org/abs/2608.11843 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3adf43aed06b (Metraipolis archive dc_3adf43aed06b, sha256 665118e819caaec5e67b2e409540ad4af2da53646f171dac100a039768c657dd)
BibTeX
@misc{dc3adf43aed06b,
  title = {When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation — arXiv:2608.11843},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.11843},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3adf43aed06b; sha256 665118e819caaec5e67b2e409540ad4af2da53646f171dac100a039768c657dd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.11843. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The Seungjeongwon Ilgi, a UNESCO Memory of the World record, is only 37.4% translated, and the most conspicuous failure mode in automatic translation is the person name -- a misread name corrupts the historical fact rather than merely the surface. Low-resource historical domains have no expert gold standard for entity translation, so practitioners substitute a knowledge base (KB) for the gold. That KB is the same resource injected into the system: scoring becomes self-referential and the metric measures instruction compliance rather than translation quality. We measure this loop. Using expert person-name annotations from the National Institute of Korean History as a gold independent of the injection pipeline, we hold the entity set fixed and vary only the provenance of the correct reading. Of 527 expert-annotated mentions, only 31.1% lie outside the injection pipeline, and the residual loop is not uniform -- in the overlapping segment the injected reading agrees with the human translation 97.8% of the time against 70.1% in the independent one, so the segment that looks healthiest is the one the loop is holding up. Across four models, a difference-in-differences analysis shows the gain from KB injection is confined to the segment whose gold shares the injected resource; in the independent segment it is at or below zero. Post-injection preservation clusters in a narrow 0.910-0.996 band even though baseline capability differs fivefold, so the reported gain is the complement of prior performance and weaker models appear to improve more dramatically. On an independent sample built by removing the construction filter, the measure replicates within model (overlapping intervals) while discriminating between models (non-overlapping intervals) -- it reflects a property of the model, not of the sample.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Digital Libraries (cs.DL).
Pagina del lavoro: https://arxiv.org/abs/2608.11843 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:15:39.220Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).