Metraipolis › archive › dc_af252e4171be
OrBIT: Structure-Guided Embedding Compression — arXiv:2610.10385
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.10385
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T12:36:26.813Z
- SHA-256 of the archived text
30e34563a94cb3e35bcf11007f060d92e432db7a74480c3509ff6c639e54395f- Stable URL of this page
https://metraipolis.com/documento/dc_af252e4171be
How to cite
OrBIT: Structure-Guided Embedding Compression — arXiv:2610.10385 — arXiv (en), https://arxiv.org/abs/2610.10385 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_af252e4171be (Metraipolis archive dc_af252e4171be, sha256 30e34563a94cb3e35bcf11007f060d92e432db7a74480c3509ff6c639e54395f)
BibTeX
@misc{dcaf252e4171be,
title = {OrBIT: Structure-Guided Embedding Compression — arXiv:2610.10385},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.10385},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_af252e4171be; sha256 30e34563a94cb3e35bcf11007f060d92e432db7a74480c3509ff6c639e54395f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10385. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Embedding tables are among the largest components of modern language models. Most compression methods fix a coding geometry such as coordinate blocks, low-rank subspaces, or unrestricted codebooks, and optimize within it. We instead ask whether the coding geometry can itself be discovered. We introduce \emph{OrBIT}, a structure-guided embedding compression framework that learns reusable local geometry from orbit dynamics and uses it to constrain a small set of shared codewords. The global reconstruction residual then decides where the fixed coding budget is spent, while redundant overlapping charts let local errors compensate one another after gluing. Our theory shows how tight-chart geometry controls distortion, how the global residual directs sequential allocation, and how data-geometry-guided refinement improves the codec. The resulting orbit machinery is compiled away, leaving a compact decoder in which the learned structure governs what is stored, where capacity is allocated, and how local information is assembled globally. Across four LLM embedding tables, OrBIT achieves $37.9\times$ compression on GPT-2 and over $23\times$ on each 7B table relative to 16-bit storage, while delivering competitive rate-distortion performance against established quantization and low-rank baselines.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.10385 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T12:14:15.344Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).