Metraipolis › archive › dc_0f1add8cedcf
Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening — arXiv:2608.05944
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.05944
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-25T21:33:07.433Z
- SHA-256 of the archived text
b1e2e343c129e6257817b0f3218bf1280fd9a0b48d8c61907d8093c7c28269d4- Stable URL of this page
https://metraipolis.com/documento/dc_0f1add8cedcf
How to cite
Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening — arXiv:2608.05944 — arXiv (en), https://arxiv.org/abs/2608.05944 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0f1add8cedcf (Metraipolis archive dc_0f1add8cedcf, sha256 b1e2e343c129e6257817b0f3218bf1280fd9a0b48d8c61907d8093c7c28269d4)
BibTeX
@misc{dc0f1add8cedcf,
title = {Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening — arXiv:2608.05944},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.05944},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0f1add8cedcf; sha256 b1e2e343c129e6257817b0f3218bf1280fd9a0b48d8c61907d8093c7c28269d4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.05944. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We report operational experience full-fine-tuning a 32.76B-parameter dense model (Qwen3-32B) on 16 x NVIDIA B300 (two nodes, FSDP / ZeRO-3) -- among the first published field accounts on this accelerator. We claim no new algorithm. The individual mechanisms we use are established practice; our contribution is the integrated field experience and a set of calibrated measurements on new hardware. Concretely we offer four practitioner artifacts. (1) A B300-calibrated power-draw triage table that distinguishes compute / communication / data-starvation / checkpoint-or-deadlock / idle by board wattage (utilization% reads 100% during an NCCL hang). (2) A set of honest negative results that dispel common optimization folklore at this scale: a controlled A/B in which per-step NFS reading matches a pretokenized local cache (~53k tok/s) because the corpus fits in page cache and the job is compute-bound; and a reconstruction of an earlier "throughput collapse" as NFS/CPU contention rather than a storage-medium limit. (3) Calibrated 4/8/16-GPU strong-scaling and GPU-hour numbers on B300 (near-linear, as expected in this regime; we report absolute values as reference data). (4) A worked failure case -- an epoch-end NCCL deadlock from per-rank token-packing imbalance -- together with a 2.7-second pre-run invariant gate and an external watcher that turn multi-hour silent failures into instant rejections. This deadlock and its remedy correspond to PyTorch's documented Join / equalize-to-minimum practice; we position our instantiation against that prior art and report the GPU-hours the failure cost and the gate saves. The transferable takeaway is operational, not algorithmic: for data-dependent data-parallel jobs, watch power rather than utilization, and verify invariants before launch -- a passing smoke test is not evidence of a safe full run. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-07 in cs.DC). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.05944 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-25T21:25:05.070Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).