Metraipolis › archive › dc_d2fb05a9140e

Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Them — arXiv:2609.05881

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.05881
License (as archived)
CC0-1.0
Archived
2026-10-16T13:58:46.907Z
SHA-256 of the archived text
297071a4e086779153dc40f5c4553f5e80bc1c4449f092f0b624477f4ace835d
Stable URL of this page
https://metraipolis.com/documento/dc_d2fb05a9140e

How to cite

Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Them — arXiv:2609.05881 — arXiv (en), https://arxiv.org/abs/2609.05881 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d2fb05a9140e (Metraipolis archive dc_d2fb05a9140e, sha256 297071a4e086779153dc40f5c4553f5e80bc1c4449f092f0b624477f4ace835d)
BibTeX
@misc{dcd2fb05a9140e,
  title = {Broken on Arrival: Silently Defective LLM Artifacts in Public Model Registries and How to Catch Them — arXiv:2609.05881},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.05881},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d2fb05a9140e; sha256 297071a4e086779153dc40f5c4553f5e80bc1c4449f092f0b624477f4ace835d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05881. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Developers increasingly run large language models locally by pulling quantized GGUF artifacts from public registries, yet nothing in the distribution pipeline functionally tests these conversions before they reach users. We executed 327 quantized code-capable model artifacts: 305 from the official Ollama library, spanning 15 model lines at every eligible quantization level at or under 8 GB, and 22 from the most-downloaded community repositories on HuggingFace. Each ran a 15-task smoke suite calibrated so that healthy artifacts pass while a known-broken one fails; suspects then faced full 164-task evaluation, a second inference backend, an independent distributor's conversion of the same model and quantization as referee, and, for community files, re-testing under the artifact's own template. The official library carries five silently defective artifacts, a batch of four Qwen2.5-Coder-3B conversions and one phi3.5-mini conversion, that solve zero of 164 tasks and zero of the smoke suite on both backends while independent conversions of the same models work: 1.6% of official artifacts, 2 of 29 model-and-size conversion groups. The adjudication chain cleared small-model artifacts that a naive threshold would condemn as broken when they are merely collapsed by extreme quantization, and it exposed two older community conversions that degrade badly on CUDA yet pass on Metal: not defective files but backend-dependent failures, a third phenomenon no registry currently tests for. Two confirmed defects produce output whose surface statistics sit inside the healthy range, invisible to any low-noise heuristic short of execution. We release the audit dataset, the quantcheck acceptance-testing tool, and disclosure reports for every confirmed defect ( this https URL ), and argue that model registries need the acceptance gate that package registries already run.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG).
Categorie arXiv: Software Engineering (cs.SE) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.05881 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:39:43.137Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).