Metraipolis › archive › dc_bd41605e9543

BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian — arXiv:2608.12894

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.12894
License (as archived)
CC0-1.0
Archived
2026-10-19T04:24:30.812Z
SHA-256 of the archived text
3d7598b6eb668d852ce6e28bbb64c5c6f2c10bd776595ccf6caa911596e11da2
Stable URL of this page
https://metraipolis.com/documento/dc_bd41605e9543

How to cite

BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian — arXiv:2608.12894 — arXiv (en), https://arxiv.org/abs/2608.12894 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bd41605e9543 (Metraipolis archive dc_bd41605e9543, sha256 3d7598b6eb668d852ce6e28bbb64c5c6f2c10bd776595ccf6caa911596e11da2)
BibTeX
@misc{dcbd41605e9543,
  title = {BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian — arXiv:2608.12894},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.12894},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bd41605e9543; sha256 3d7598b6eb668d852ce6e28bbb64c5c6f2c10bd776595ccf6caa911596e11da2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12894. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Cultural evaluation of large language models (LLMs) often focuses on high-resource standard languages, leaving regional culture and dialect communities underrepresented. We introduce BavGround, a benchmark for evaluating Bavarian regional cultural grounding and dialect competence across English, German and Bavarian. BavGround contains 206 multiple-choice source questions across eight cultural domains per language, yielding 618 multi-parallel instances, with items covering both broadly accessible cultural knowledge and source-grounded regional knowledge from journalism, historical sources, and specialist literature. We evaluate fifteen 7B-10B open-weight instruction-tuned models and one closed-model reference. Strong multilingual models perform best overall, but performance drops on Bavarian items and source-grounded questions, indicating persistent difficulty with dialectal and localized cultural knowledge. We further show that conclusions depend strongly on evaluation protocol: raw answer-letter scoring, shuffled-letter scoring, option-text likelihood, generated-answer parsing, and semantic matching can produce different absolute scores and rankings, especially for regionally adapted models. Finally, an exploratory analysis of GENBA-10B checkpoints suggests that continued pretraining improves answer-content likelihood unevenly across domains, while dialect competence remains comparatively weak. BavGround supports localized, protocol-aware evaluation of cultural representation in LLMs.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-14 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.12894 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T04:17:42.384Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).