Metraipolis › archive › dc_ad7ee3bf0deb

FreqBLiMP: Frequency-Controlled Minimal Pairs Reveal Robustness and Fragility of LLMs Under Lexical Rarity — arXiv:2609.07153

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.07153
License (as archived): CC0-1.0
Archived: 2026-10-16T12:45:28.691Z
SHA-256 of the archived text: d4401a3d3b0e2ebed65a0708807eb6f93fd6911e050b644f4b9ee25974dddaf2
Stable URL of this page: https://metraipolis.com/documento/dc_ad7ee3bf0deb

How to cite

FreqBLiMP: Frequency-Controlled Minimal Pairs Reveal Robustness and Fragility of LLMs Under Lexical Rarity — arXiv:2609.07153 — arXiv (en), https://arxiv.org/abs/2609.07153 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ad7ee3bf0deb (Metraipolis archive dc_ad7ee3bf0deb, sha256 d4401a3d3b0e2ebed65a0708807eb6f93fd6911e050b644f4b9ee25974dddaf2)

BibTeX

@misc{dcad7ee3bf0deb,
  title = {FreqBLiMP: Frequency-Controlled Minimal Pairs Reveal Robustness and Fragility of LLMs Under Lexical Rarity — arXiv:2609.07153},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.07153},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ad7ee3bf0deb; sha256 d4401a3d3b0e2ebed65a0708807eb6f93fd6911e050b644f4b9ee25974dddaf2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.07153. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Minimal-pair benchmarks such as BLiMP evaluate linguistic knowledge by testing whether language models (LMs) prefer acceptable sentences over minimally different unacceptable ones. However, these benchmarks largely ignore lexical frequency variation, despite lexical frequency being a pervasive and highly skewed property of natural language use. Consequently, existing evaluations do not test whether grammatical preferences remain stable when contrasts involve rare lexical items. We introduce FreqBLiMP, a frequency-controlled extension of BLiMP that regenerates all 67 paradigms under explicit Zipf-frequency regimes while preserving each minimal-pair's grammatical contrast. Evaluating multiple open-weight LLM families across scales, we find that decreasing lexical frequency produces a consistent, monotonic decrease in sentence likelihood, but only a modest reduction in overall contrastive acceptability accuracy. However, this aggregate stability masks substantial variation across linguistic phenomena, with LLMs remaining robust on overt morphosyntactic generalization while degrading on phenomena that require lemma-specific information.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.07153 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T12:30:25.098Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).