Metraipolis › archive › dc_caa493e24d8b
To Explore The Strange New World Beyond Data Distribution: System Behavior, Causality Tax, and Non-causal Base Model — arXiv:2610.02839
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.02839
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-05T22:51:57.079Z
- SHA-256 of the archived text
881c5f585e79b67499a92e60b35519ebd4333094a4c8277546410c2223e6ca95- Stable URL of this page
https://metraipolis.com/documento/dc_caa493e24d8b
How to cite
To Explore The Strange New World Beyond Data Distribution: System Behavior, Causality Tax, and Non-causal Base Model — arXiv:2610.02839 — arXiv, https://arxiv.org/abs/2610.02839 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_caa493e24d8b (Metraipolis archive dc_caa493e24d8b, sha256 881c5f585e79b67499a92e60b35519ebd4333094a4c8277546410c2223e6ca95)
BibTeX
@misc{dccaa493e24d8b,
title = {To Explore The Strange New World Beyond Data Distribution: System Behavior, Causality Tax, and Non-causal Base Model — arXiv:2610.02839},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.02839},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_caa493e24d8b; sha256 881c5f585e79b67499a92e60b35519ebd4333094a4c8277546410c2223e6ca95},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02839. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We show that the causality of language models (LMs) may not be necessary nor optimal. This is the case when system behavior (denoted as $S$) is incorporated as a first-principle Bayesian feature. Here, $S$ refers to extra dominant factors beyond the data space, and they involve coupled effects. Despite being the de facto foundation of modern architecture, recent studies indicate persistent mismatches and contradictions with causality. These issues largely stem from system behavior rather than the data distribution. We therefore propose the SBD framework, which incorporates $S$ as an irreducible component of the evidence lower bound (ELBO). SBD theoretically reveals a counter-intuitive Causality Tax phenomenon, where causality emerges as a suboptimal approximation with an additional structural error, due to the obliviousness to $S$. To address the challenge of latent variable analysis, we validate the SBD-predicted impact of $S$ via implicit measurements, theoretical-bound-guided controls, and Neural Tangent Kernel (NTK) evaluations. In particular, we construct Green Shell (GSH) to show the possibility of reducing Causality Tax. GSH is a non-causal variational family, and it replaces the sequential dependency chain of $S$ components with a divide-and-conquer partition. NTK spectra in the lazy-training regime confirm that GSH always achieves significantly tighter error bounds than causality, with $7dB+$ improvement in signal-to-noise ratio. In the relatively later stage of lazy-training, GSH further leads to superior generalization (up to $20\%$ richer multi-scale fitting capabilities). Taken together, SBD establishes system behavior as a complementary theoretical abstraction besides causality and distribution fitting, opening new research avenues such as designing and optimizing LM base models. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.02839 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-05T22:46:10.615Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).