Metraipolis › archive › dc_e968d98ba8a4
Lost in Rounding: bf16 Recurrent State Makes Hybrid Reasoning Models Think Longer — arXiv:2611.12127
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.12127
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-10T18:43:55.250Z
- SHA-256 of the archived text
8b8922ca8f0fe7f7485f9ecae6e098b4f6866cb38da79338e0a4ccd796dae73b- Stable URL of this page
https://metraipolis.com/documento/dc_e968d98ba8a4
How to cite
Lost in Rounding: bf16 Recurrent State Makes Hybrid Reasoning Models Think Longer — arXiv:2611.12127 — arXiv (en), https://arxiv.org/abs/2611.12127 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e968d98ba8a4 (Metraipolis archive dc_e968d98ba8a4, sha256 8b8922ca8f0fe7f7485f9ecae6e098b4f6866cb38da79338e0a4ccd796dae73b)
BibTeX
@misc{dce968d98ba8a4,
title = {Lost in Rounding: bf16 Recurrent State Makes Hybrid Reasoning Models Think Longer — arXiv:2611.12127},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.12127},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e968d98ba8a4; sha256 8b8922ca8f0fe7f7485f9ecae6e098b4f6866cb38da79338e0a4ccd796dae73b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.12127. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Hybrid reasoning models interleave attention with recurrent mixers such as Mamba-2 and Gated DeltaNet, whose state is cached between decoding steps. Prior work showed that storing this state in FP16 or INT8 with round-to-nearest (RTN) makes Mamba-2 and GDN hybrids more verbose, and that stochastic rounding (SR) fixes it. We study the other 16-bit format that serving engines expose, bf16, on six open hybrid reasoning models and 590 competition-math problems. With bf16 RTN state, five of six models reason longer (geometric-mean length 1.07x to 1.27x; also under greedy decoding in the two models tested) and hit the 32K token cap more often. On Qwen3.5-9B, cap hits rise from 118 to 223 and accuracy falls from 85.6% to 70.8%. On LiveCodeBench, both models tested also reason longer and lose 12.6 and 13.4 points of pass@1. Rerunning the capped items with a 128K budget halves the accuracy gap but does not close it (93.6% vs. 86.6%): bf16 runs that never finish are twice as common, and 85% of them are exact repetition loops. On Nemotron-Nano-9B-v2, by contrast, 30 of 31 extra caps finish at 128K, so the cost there is mostly extra tokens (exploratory). Same-width bf16 SR removes the length increase in four of five affected models, restores Qwen3.5-9B to 87.6% and brings code length and pass@1 near FP32 levels, so the failure is mostly rounding bias, not missing resolution. With an integer-hash random generator, SR on Qwen3.5-9B matches FP32 on cap hits and accuracy (113 vs. 118; 85.6% for both) at the memory and decoding speed of bf16 RTN. In the Nemotron-Nano-v2 lineage, SR on 25% of heads (an intermediate-decay band) removes the effect, while random heads and the top teacher-forced-KL octiles do not; in Nemotron-3 and Qwen3.5-9B the effect is spread across heads. We recommend that evaluations of hybrid models report the state dtype, cap rate and output length alongside accuracy. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.12127 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-10T18:37:37.678Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).