Metraipolis › archive › dc_dfe1099cfac2
Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionality — arXiv:2609.34936
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34936
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:53:34.547Z
- SHA-256 of the archived text
5b9ed8183f78acd9e15d55b62be73836b6f5a3f719ffa132cd0e2a4d91542c5e- Stable URL of this page
https://metraipolis.com/documento/dc_dfe1099cfac2
How to cite
Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionality — arXiv:2609.34936 — arXiv (en), https://arxiv.org/abs/2609.34936 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dfe1099cfac2 (Metraipolis archive dc_dfe1099cfac2, sha256 5b9ed8183f78acd9e15d55b62be73836b6f5a3f719ffa132cd0e2a4d91542c5e)
BibTeX
@misc{dcdfe1099cfac2,
title = {Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionality — arXiv:2609.34936},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34936},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dfe1099cfac2; sha256 5b9ed8183f78acd9e15d55b62be73836b6f5a3f719ffa132cd0e2a4d91542c5e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34936. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
It is unclear how Neural Language Models (NLMs) acquire the structural meaning encoded by grammatical structures that is independent of lexical semantics. We propose a statistical learning process in which learned dependency structures themselves become new distributional units for subsequent statistical learning. Under this account, once a dependency structure is acquired, the model tracks its contextual distributions. These contextual features reflect the semantic properties of a composite structure. To test this hypothesis, we design a synthetic grammar in which each grammatical structure has distinct contextual distributions that cannot be recovered from the distributional statistics of their component tokens alone. We train a series of BERT-style masked language models on this grammar and examine their developmental trajectory. The results show that models can successfully learn the contextual distributions of composite dependency structures even though they cannot be inferred from token statistics alone. Developmental analysis further reveals a clear developmental trajectory. The learning of the dependency relations that define a grammatical structure consistently precedes the learning of its contextual features. These findings suggest that statistical learning in NLMs is not merely the accumulation of token co-occurrence statistics, but a process in which learned dependency structures become new units of distributional learning. We argue that this process provides a statistical-learning account of how NLMs solve the compositionality problem in language. Finally, we discuss the possibility that this statistical learning process provides an explanatory theory on how language cognition could emerge from pure distributional statistics. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.34936 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T21:27:08.648Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).