Metraipolis › archive › dc_3d57cfcc9505

The Emergence and Sparsification of a Syntactic Circuit — arXiv:2610.33299

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.33299
License (as archived)
CC0-1.0
Archived
2026-10-27T08:11:12.413Z
SHA-256 of the archived text
74d867b14b2f92fc95210327158d98c397a833ee5cb7144b7ab6d135dbc01a8e
Stable URL of this page
https://metraipolis.com/documento/dc_3d57cfcc9505

How to cite

The Emergence and Sparsification of a Syntactic Circuit — arXiv:2610.33299 — arXiv (en), https://arxiv.org/abs/2610.33299 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3d57cfcc9505 (Metraipolis archive dc_3d57cfcc9505, sha256 74d867b14b2f92fc95210327158d98c397a833ee5cb7144b7ab6d135dbc01a8e)
BibTeX
@misc{dc3d57cfcc9505,
  title = {The Emergence and Sparsification of a Syntactic Circuit — arXiv:2610.33299},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.33299},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3d57cfcc9505; sha256 74d867b14b2f92fc95210327158d98c397a833ee5cb7144b7ab6d135dbc01a8e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.33299. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds of pythia-410m across training. We find that subject-verb agreement is learned early, and is initially handled by a diffuse collection of neurons. However, after agreement behavior has stabilized, the mechanisms responsible for it continue to reorganize. The attention heads which mediate agreement often shift during training: in the case of pythia-1b, behavior is controlled by two heads early in training but by a third (different) head later. Furthermore, the circuit shrinks over training, with pythia-1b's final circuit being strikingly sparse. We take these results to be informative about how linguistic knowledge emerges and is represented in a language model.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.33299 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:56:00.189Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).