Metraipolis › archive › dc_28a5f9734dca
When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior — arXiv:2610.03057
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.03057
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-05T23:10:05.352Z
- SHA-256 of the archived text
5e1538e960e74c4eb7148c7edbd563819b3edd4e416f95cbede93b7cae719625- Stable URL of this page
https://metraipolis.com/documento/dc_28a5f9734dca
How to cite
When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior — arXiv:2610.03057 — arXiv, https://arxiv.org/abs/2610.03057 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_28a5f9734dca (Metraipolis archive dc_28a5f9734dca, sha256 5e1538e960e74c4eb7148c7edbd563819b3edd4e416f95cbede93b7cae719625)
BibTeX
@misc{dc28a5f9734dca,
title = {When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior — arXiv:2610.03057},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.03057},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_28a5f9734dca; sha256 5e1538e960e74c4eb7148c7edbd563819b3edd4e416f95cbede93b7cae719625},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03057. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Relational foundation models are increasingly pretrained on synthetic databases, yet downstream benchmarks reveal little about why one synthetic corpus produces a better model than another. In particular, strong performance may arise from realistic row-level statistics without the model ever learning to use relational structure. We study this as a data-attribution problem: which property of synthetic pretraining data induces relational computation? Using four Relational Transformer checkpoints trained with the same architecture, initialization, objective, and compute budget on corpora produced by four relational data generators, we trace a measurable property of the data to learned computation and downstream behavior. We hypothesize that relational mechanisms emerge when cross-table information is predictively necessary for the masked-cell pretraining objective. RelDiff exhibits by far the largest predictive gain from foreign-key-linked parents, and its corresponding model is uniquely sensitive to foreign-key interventions on unseen databases. This dependence survives a random-initialization control, grows monotonically with the fraction of corrupted links, and localizes to a serial cross-table pathway. Finally, disrupting the same mechanism during downstream inference removes RelDiff's advantage on relational tasks while leaving structure-insensitive models nearly unchanged. These results connect a property of synthetic training data to a learned mechanism and, through intervention, to downstream behavior. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.03057 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-05T23:01:38.665Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).