Metraipolis › archive › dc_f71fa7ac98e0

FFSlim: An Efficient and Lightweight Format for Multi-modal Data Storage and Retrieval — arXiv:2608.27865

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27865
License (as archived)
CC0-1.0
Archived
2026-10-17T10:56:57.221Z
SHA-256 of the archived text
2524da73552e459d3694989da6b9c04b3ed5e798be48e845402d4c92375dfc1c
Stable URL of this page
https://metraipolis.com/documento/dc_f71fa7ac98e0

How to cite

FFSlim: An Efficient and Lightweight Format for Multi-modal Data Storage and Retrieval — arXiv:2608.27865 — arXiv (en), https://arxiv.org/abs/2608.27865 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f71fa7ac98e0 (Metraipolis archive dc_f71fa7ac98e0, sha256 2524da73552e459d3694989da6b9c04b3ed5e798be48e845402d4c92375dfc1c)
BibTeX
@misc{dcf71fa7ac98e0,
  title = {FFSlim: An Efficient and Lightweight Format for Multi-modal Data Storage and Retrieval — arXiv:2608.27865},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27865},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f71fa7ac98e0; sha256 2524da73552e459d3694989da6b9c04b3ed5e798be48e845402d4c92375dfc1c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27865. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

With the rapid expansion of large-scale media-text corpora, multi-modal datasets increasingly require efficient storage and retrieval. Existing formats such as Files, TDP, and FFRecord work adequately for uni-modal data but expose fundamental limitations in multi-modal settings, including storage redundancy, massive small-file overheads, cache-unfriendly layouts, and heavy index structures. These issues jointly inflate storage and memory usage and make I/O the dominant bottleneck in real training workloads. We present FFSlim, a lightweight format for storing and retrieving multi-modal data. FFSlim improves storage efficiency and loading throughput through three components: a unified file format that removes media duplication and avoids small-file proliferation; an adaptive retrieval mechanism that enables low-overhead pair-level access and accelerates repeated media loading; and a redundancy detection and aggregation module that converts existing datasets into the FFSlim layout. The experimental results demonstrate that FFSlim achieves 2.07x and 8.26x higher data loading and write throughput on average than the strongest baseline, with minimal storage and index overhead. Consequently, these underlying I/O accelerations enable FFSlim to reduce end-to-end training time by 5.36%-14.18% across seven diverse multi-modal models.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-31 in cs.PF).
Categorie arXiv: Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2608.27865 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:48:20.371Z dalla pagina https://arxiv.org/catchup/cs.PF/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).