Metraipolis › archive › dc_5db2df1512ac
Measuring Stability and Failure Behavior in Language Models Under Structured Perturbations — arXiv:2608.22138
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.22138
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T07:41:17.530Z
- SHA-256 of the archived text
13b680d6ab6d9e47530fcb6be650bb8f983e76eec0b4b4dce6236532f213c4d1- Stable URL of this page
https://metraipolis.com/documento/dc_5db2df1512ac
How to cite
Measuring Stability and Failure Behavior in Language Models Under Structured Perturbations — arXiv:2608.22138 — arXiv (en), https://arxiv.org/abs/2608.22138 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5db2df1512ac (Metraipolis archive dc_5db2df1512ac, sha256 13b680d6ab6d9e47530fcb6be650bb8f983e76eec0b4b4dce6236532f213c4d1)
BibTeX
@misc{dc5db2df1512ac,
title = {Measuring Stability and Failure Behavior in Language Models Under Structured Perturbations — arXiv:2608.22138},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.22138},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5db2df1512ac; sha256 13b680d6ab6d9e47530fcb6be650bb8f983e76eec0b4b4dce6236532f213c4d1},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.22138. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Language models are usually judged by a single accuracy score, which does not reveal how their performance degrades as inputs are perturbed. We present a graded, multi-family, failure-aware framework for stress-testing reasoning models. It perturbs each problem along a multi-level severity ladder across seven families: six that preserve the answer, paraphrase, input noise, formatting, irrelevant context, context load, and conflicting instructions, and a Knowledge Boundary family that removes answerability so that refusal becomes the correct response. Every test is validity-gated and labeled by its measured severity, and each model is summarized by per-level Accuracy, a magnitude-weighted Stability, and a per-family Collapse Point defined relative to the model's own baseline. Instantiated on the same 100 seed problems used by GSM-Symbolic, expanded into 4,473 gated tests and run on four models spanning capability tiers, the framework exposes structure that an aggregate score hides: the level at which a model fails is family-specific rather than global, and two stressors expose consistent weaknesses across all models: conflicting instructions and questions built on an impossible premise. Recognition of unanswerability is otherwise uneven, reliable on missing information and fabricated evidence but weak on impossible premises. These failure points are invisible to standard accuracy reporting. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-25 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.22138 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T07:37:52.704Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).