Metraipolis › archive › dc_d47145fded84
TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translation — arXiv:2609.18156
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.18156
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T13:49:21.301Z
- SHA-256 of the archived text
def87a78d99e840e81692a63c74383f78409f14d36015b10c0e3162cf5371839- Stable URL of this page
https://metraipolis.com/documento/dc_d47145fded84
How to cite
TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translation — arXiv:2609.18156 — arXiv (en), https://arxiv.org/abs/2609.18156 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d47145fded84 (Metraipolis archive dc_d47145fded84, sha256 def87a78d99e840e81692a63c74383f78409f14d36015b10c0e3162cf5371839)
BibTeX
@misc{dcd47145fded84,
title = {TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translation — arXiv:2609.18156},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.18156},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d47145fded84; sha256 def87a78d99e840e81692a63c74383f78409f14d36015b10c0e3162cf5371839},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18156. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating large language models remain limited. We present TeochewBench, a human-reviewed benchmark comprising 300 Teochew Hanzi expressions for evaluating translation from Teochew Hanzi into Mandarin Chinese and English. The dataset covers five categories: basic vocabulary; everyday sentences; Teochew-specific expressions; tone, politeness, and context; and idiomatic, ambiguous, and culturally specific expressions. A primary Teochew-speaking reviewer examined all entries individually and revised them as needed, while two additional Teochew speakers verified selected items. Our main evaluation covers 11 official general-purpose post-trained models on the reviewed dataset in both translation directions, yielding 6,600 predictions. Two official base checkpoints provide 1,200 predictions for supplementary diagnostics, bringing the total to 13 models and 7,800 predictions. We additionally include a Hanzi-copy control, which returns the source input unchanged, to assess how shared Hanzi affect automatic scores for translation into Mandarin Chinese. Qwen3.5-27B achieved the highest overall chrF-style score among the evaluated checkpoints, at 60.63, followed by Qwen2.5-72B-Instruct at 56.61, Gemma-3-27B-IT at 56.36, and GLM-4-32B-0414 at 55.82. Across the 11 main-evaluation models, the mean chrF-style score decreased from 69.25 for low-specificity items to 27.52 for high-specificity items. High-specificity expressions received lower scores and exhibited smaller cross-model differences, suggesting that they constitute a shared low-scoring region across the model families evaluated here. The Hanzi-copy control further indicates that surface overlap in low-specificity items can substantially affect automatic scores for translation into Mandarin Chinese. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-17 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.18156 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T13:48:24.300Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).