Metraipolis › archive › dc_40ac3de1eec3

Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants — arXiv:2610.33493

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.33493
License (as archived)
CC0-1.0
Archived
2026-10-27T08:11:20.030Z
SHA-256 of the archived text
a4bd5e4a07053fb15f3b43e9f9b1fb6a0c47259a646c660cbe58f98e9785d0d5
Stable URL of this page
https://metraipolis.com/documento/dc_40ac3de1eec3

How to cite

Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants — arXiv:2610.33493 — arXiv (en), https://arxiv.org/abs/2610.33493 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_40ac3de1eec3 (Metraipolis archive dc_40ac3de1eec3, sha256 a4bd5e4a07053fb15f3b43e9f9b1fb6a0c47259a646c660cbe58f98e9785d0d5)
BibTeX
@misc{dc40ac3de1eec3,
  title = {Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants — arXiv:2610.33493},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.33493},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_40ac3de1eec3; sha256 a4bd5e4a07053fb15f3b43e9f9b1fb6a0c47259a646c660cbe58f98e9785d0d5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.33493. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

French users may omit apostrophes and diacritics or make keyboard typos in everyday input. The effects of these common variations on language model performance and prediction reliability remain insufficiently understood. To address this gap, we construct a controlled French surface-variant benchmark covering seven task types, with 4,200 original-low-high triplets (12,600 input instances). Each high-intensity variant retains all low-intensity edits and adds further changes. We evaluate six models from two LLM families. The results show that accuracy decreases and Brier score and negative log-likelihood (NLL) increase with perturbation intensity. Larger models show smaller accuracy losses under high intensity, but increases in Brier score and NLL do not consistently decrease with scale. Keyboard typos cause the largest losses in accuracy and the quality of predicted probabilities at the predefined high-intensity settings. Finally, calibration temperatures fitted on original inputs transfer effectively to surface variants. Combining calibration with input restoration further improves Brier score and NLL over either method alone while preserving restored accuracy. Our benchmark provides a unified framework for jointly evaluating task accuracy and prediction reliability under common French surface variations.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.33493 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:56:00.198Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).