Metraipolis › archive › dc_f0a5b074b034
Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility — arXiv:2608.03930
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.03930
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:19:42.171Z
- SHA-256 of the archived text
d117e5be812c5c0758a4fbf5f554c18d74ae96568549acccd9968871fbd72c0c- Stable URL of this page
https://metraipolis.com/documento/dc_f0a5b074b034
How to cite
Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility — arXiv:2608.03930 — arXiv (en), https://arxiv.org/abs/2608.03930 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f0a5b074b034 (Metraipolis archive dc_f0a5b074b034, sha256 d117e5be812c5c0758a4fbf5f554c18d74ae96568549acccd9968871fbd72c0c)
BibTeX
@misc{dcf0a5b074b034,
title = {Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility — arXiv:2608.03930},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.03930},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f0a5b074b034; sha256 d117e5be812c5c0758a4fbf5f554c18d74ae96568549acccd9968871fbd72c0c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03930. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Pre-pretraining language models (LMs) on symbolic data can accelerate and improve natural language acquisition. However, existing pre-pretraining tasks, such as Dyck and procedural algorithms, rely on narrow primitives that fail to capture the expressive capacity of natural language. Moreover, prior studies remain restricted to relatively small token budgets, offering limited insight into skill emergence and representational dynamics. To address these limitations, we propose logic pre-pretraining (Logic-PPT) as a principled initialization strategy, leveraging formal derivations to impart richer structural and linguistic biases. Formal derivations require abstract mechanisms that are central to natural language, simultaneously binding variables, connecting quantifiers and relational dependencies, and composing predicate-argument structures over long contexts. Scaling our evaluation to a 100B-token regime, logic pre-pretraining substantially accelerates skill acquisition in LMs, achieving 80\% accuracy on linguistic tasks with 36B fewer tokens than standard initialization, and outperforming alternative pre-pretraining baselines. Mechanistically, formal derivations induce persistent structural reorganization, distinctively characterized by a lower-rank, spectrally concentrated representation space. Crucially, we show that this internal geometry enables improved model compressibility via pruning, matching the dense baseline performance even at $\approx$33\% sparsity. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.03930 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T12:05:53.102Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).