Metraipolis › archive › dc_ef24a8eb7757

Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks — arXiv:2609.34800

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.34800
License (as archived)
CC0-1.0
Archived
2026-10-07T21:53:32.421Z
SHA-256 of the archived text
05043367413d1de6d1161f6e6fd267691ea00f1039c02b320543e379c9b6e63b
Stable URL of this page
https://metraipolis.com/documento/dc_ef24a8eb7757

How to cite

Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks — arXiv:2609.34800 — arXiv (en), https://arxiv.org/abs/2609.34800 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ef24a8eb7757 (Metraipolis archive dc_ef24a8eb7757, sha256 05043367413d1de6d1161f6e6fd267691ea00f1039c02b320543e379c9b6e63b)
BibTeX
@misc{dcef24a8eb7757,
  title = {Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks — arXiv:2609.34800},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34800},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ef24a8eb7757; sha256 05043367413d1de6d1161f6e6fd267691ea00f1039c02b320543e379c9b6e63b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34800. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The rapid advancement of Large Language Models (LLMs) imposes a thorough evaluation of their linguistic and analytical capabilities as well as constraints, particularly for a language with limited benchmark coverage such as Greek. To address the limited availability of comprehensive benchmarks in this domain, we introduce Prot-Ex and Pan-Ex, two benchmarks consisting of questions from entrance exams for Greek Model and Experimental schools as well as the Panhellenic exams (the Greek national university entrance examinations). These benchmarks are employed to assess the performance of text-only LLMs-including the Greek-adapted KriKri-8B-Instruct, Llama-3.1-8B, Gemma-4-26B, and Qwen-3-32B-across diverse academic disciplines (Modern Greek, Mathematics, Physics, etc.) and task formats (closed, structured, and open-ended), including textualized visual context (i.e., image descriptions). Our findings indicate the localized KriKri-8B significantly outperforms its base model, successfully rivalling much larger LLMs in linguistically demanding humanities tasks. By leveraging an LLM-as-a-Judge methodology, we expose the inadequacy of traditional lexical metrics for evaluating complex reasoning. Crucially, we uncover a few-shot prompting paradox: while synthetic examples improve accuracy in closed-ended questions, they severely overload the context window of 8B models in structured tasks, causing significant performance degradation. Ultimately, this study suggests targeted linguistic adaptation offsets lower parameter counts in specialized domains, despite the fragility of smaller models to prompt verbosity.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.34800 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T21:27:08.640Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).