Metraipolis › archive › dc_edbecc1d90ac
Sensitivity Inflation from Cross-Record Precision Routing in Private Quantized Fine-Tuning — arXiv:2611.01411
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.01411
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:36:52.864Z
- SHA-256 of the archived text
678735e8c288e596e4a04113110293f4f9c86bf67687cad27b1578b1c534a134- Stable URL of this page
https://metraipolis.com/documento/dc_edbecc1d90ac
How to cite
Sensitivity Inflation from Cross-Record Precision Routing in Private Quantized Fine-Tuning — arXiv:2611.01411 — arXiv (en), https://arxiv.org/abs/2611.01411 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_edbecc1d90ac (Metraipolis archive dc_edbecc1d90ac, sha256 678735e8c288e596e4a04113110293f4f9c86bf67687cad27b1578b1c534a134)
BibTeX
@misc{dcedbecc1d90ac,
title = {Sensitivity Inflation from Cross-Record Precision Routing in Private Quantized Fine-Tuning — arXiv:2611.01411},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.01411},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_edbecc1d90ac; sha256 678735e8c288e596e4a04113110293f4f9c86bf67687cad27b1578b1c534a134},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01411. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Private fine-tuning of large language models increasingly pairs DP-SGD with low-rank adapters on frozen, quantized backbones, and every such stack inherits one unstated premise: frozen layers keep each record's clipped gradient a function of that record alone. We show that LLM.int8 breaks this premise. It routes outlier activation columns to a higher-precision path using a union over the whole batch, so one record decides how every other record's activations are computed, even though the weights are frozen and the quantization scales are per row. We bound the resulting sensitivity inflation, state a record-isolation contract that restores the standard bound, and define record-owned precision dispatch that meets it with the same GEMMs as batched LLM.int8, so isolation needs no extra arithmetic. We then audit the native CUDA stack (bitsandbytes, PEFT, Opacus) against four matched controls. A constructed LoRA-Opacus witness reaches $15.0\times$ the add/remove bound. On pretrained OPT-125M with natural text, other records alter 20% of record-layer precision splits at batch size 16. An exploratory re-clipping analysis finds 10 of 64 candidate neighbors above the bound once every gradient is clipped; the top pair, revalidated natively, reaches $1.065C$ while every control stays within $C$, and forcing one layer's outlier mask removes the excess. A two-million-draw audit of the released adapters yields $\varepsilon$ lower bounds ($3.25$ at $\delta=10^{-5}$) below the advertised guarantee ($4.38$). Resolving an excess this small at the release level would take about 200 times more draws, whereas checking the sensitivity contract directly exposes it in a few native runs, which makes contract checking the practical audit for private quantized training.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.01411 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:12:54.498Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).