Metraipolis › archive › dc_dbeaaea67a9e
Capability-Stratified Degradation in Ternary Language Models — arXiv:2608.28809
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.28809
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T07:43:51.552Z
- SHA-256 of the archived text
5cbffb26009f7fc2058f867ffc59e8955762c5fb8057f566e8fe95de1522a1ba- Stable URL of this page
https://metraipolis.com/documento/dc_dbeaaea67a9e
How to cite
Capability-Stratified Degradation in Ternary Language Models — arXiv:2608.28809 — arXiv (en), https://arxiv.org/abs/2608.28809 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dbeaaea67a9e (Metraipolis archive dc_dbeaaea67a9e, sha256 5cbffb26009f7fc2058f867ffc59e8955762c5fb8057f566e8fe95de1522a1ba)
BibTeX
@misc{dcdbeaaea67a9e,
title = {Capability-Stratified Degradation in Ternary Language Models — arXiv:2608.28809},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.28809},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dbeaaea67a9e; sha256 5cbffb26009f7fc2058f867ffc59e8955762c5fb8057f566e8fe95de1522a1ba},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28809. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to $\{-1,0,+1\}$, approaching the limit of $\log_2 3 \approx 1.585$ bits/weight. The practical question for a pretrained model is not simply whether weights can be quantised but which capabilities survive and whether it remains useful for adaptation. We explore this by converting Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT). The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning. The evidence shows non-uniform degradation. A linear probe recovers 43.76% of MMLU answers from the full-precision teacher's representations but only 26.19% from Cloe (near chance), indicating specialist factual information is lost. However, Cloe retains measurable performance on ten tasks, averaging 77.1% of teacher performance. Crucially, fine-tuning raises Cloe to 89.8% on SST-2 (95.6% of the matched teacher) and reaches 79.4% teacher retention on XSum. We attribute degradation to a combination of quantisation-induced information loss and incomplete recovery due to the limited QAT budget. We also highlight an evaluation pitfall: standard answer-letter scoring failed (Cloe emitted "A" on 98.6% of MMLU questions), necessitating continuation scoring. Ultimately, ternary conversion is unsuitable as a drop-in general replacement yet remains valuable as a compact substrate for task-specific models.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.28809 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.482Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).