Metraipolis › archive › dc_fcfe2ebf5be4

Structure, Dimensionality, and Distribution Shape - What Makes Pretraining Data Transferable, and Why the Input Pipeline Decides Which — arXiv:2611.04106

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.04106
License (as archived): CC0-1.0
Archived: 2026-11-04T18:54:00.570Z
SHA-256 of the archived text: b91e8f1d4e75c0980d26d8e93f4464360454e10eec9921ecc3672948d4716404
Stable URL of this page: https://metraipolis.com/documento/dc_fcfe2ebf5be4

How to cite

Structure, Dimensionality, and Distribution Shape - What Makes Pretraining Data Transferable, and Why the Input Pipeline Decides Which — arXiv:2611.04106 — arXiv (en), https://arxiv.org/abs/2611.04106 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fcfe2ebf5be4 (Metraipolis archive dc_fcfe2ebf5be4, sha256 b91e8f1d4e75c0980d26d8e93f4464360454e10eec9921ecc3672948d4716404)

BibTeX

@misc{dcfcfe2ebf5be4,
  title = {Structure, Dimensionality, and Distribution Shape - What Makes Pretraining Data Transferable, and Why the Input Pipeline Decides Which — arXiv:2611.04106},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04106},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fcfe2ebf5be4; sha256 b91e8f1d4e75c0980d26d8e93f4464360454e10eec9921ecc3672948d4716404},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04106. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

What makes a pretraining corpus transferable? We describe a corpus by three properties: its structure, its dimensionality, and the distribution shape of each unit's values. We propose the Transfer-Invariance Principle: the same property can be irrelevant to transfer under one input transform and decisive under another, so the model's input transform, not the data modality, decides which properties can matter. To test it we build TransferForge, a generator that places a synthetic population at any prescribed structure, dimensionality, and distribution shape. KernelSynth, the synthetic prior of Chronos, cannot, because it draws independent series. When the model standardizes each input window (PatchTST on UCR), the distribution shape is inert and the structural setting governs. Synthetic pretraining then matches a real corpus of the same size and exceeds it once scaled. When the model encodes calcium fluorescence signals without normalization (POYO Plus), the shape becomes a governing constraint, and a shape-matched synthetic corpus replaces real pretraining with zero real neural activity. Flipping only the input normalization, with the model and data fixed, changes the effect of a shape manipulation by a factor of 5.3, so the pipeline, not the modality, sets the switch. The principle yields a recipe that chooses what to synthesize before any training. Pretraining data then becomes something to manufacture for the model, most valuably where real data is scarce, expensive, or private.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.04106 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:35:37.427Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).