Metraipolis › archive › dc_4eed84b83409
FrameBench:A Language Understanding Benchmark Based on Frame Semantics — arXiv:2609.03370
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.03370
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T01:40:22.580Z
- SHA-256 of the archived text
20e2c9c17d7d9c29e676409e79267a23f595489e9b705f26f91f671568867929- Stable URL of this page
https://metraipolis.com/documento/dc_4eed84b83409
How to cite
FrameBench:A Language Understanding Benchmark Based on Frame Semantics — arXiv:2609.03370 — arXiv (en), https://arxiv.org/abs/2609.03370 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4eed84b83409 (Metraipolis archive dc_4eed84b83409, sha256 20e2c9c17d7d9c29e676409e79267a23f595489e9b705f26f91f671568867929)
BibTeX
@misc{dc4eed84b83409,
title = {FrameBench:A Language Understanding Benchmark Based on Frame Semantics — arXiv:2609.03370},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.03370},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4eed84b83409; sha256 20e2c9c17d7d9c29e676409e79267a23f595489e9b705f26f91f671568867929},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03370. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
In frame semantics, sentence comprehension is assumed to proceed by relating lexical meaning to background knowledge called semantic frames, thereby enabling readers to implicitly enrich the text with unstated information. Recent large language models (LLMs) have achieved strong performance across a wide range of downstream tasks. However, it remains unclear whether they can reproduce the kinds of implicit enrichment that humans naturally make during comprehension. To address this question, we introduce FrameBench, a benchmark grounded in frame semantics. FrameBench consists of multiple-choice questions that test whether models distinguish the frames evoked by the same verb across contexts. We construct the benchmark for English and Japanese using FrameNet-style resources and a generation-and-verification pipeline with native-speaker judgments. Our experiments on a diverse set of models reveal challenges for small models, while several large models surpass the human reference scores. We release the constructed FrameBench dataset and the code for dataset construction and evaluation at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.03370 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T01:11:43.526Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).