Metraipolis › archive › dc_eccaf7d76889

Graph Machine: Towards Better Pretraining via Edges — arXiv:2609.02881

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.02881
License (as archived)
CC0-1.0
Archived
2026-10-17T05:00:43.766Z
SHA-256 of the archived text
5a39bbf17453ec14a7c0b65d8c430d740b874de628ceccf6c0be1d051916fb3a
Stable URL of this page
https://metraipolis.com/documento/dc_eccaf7d76889

How to cite

Graph Machine: Towards Better Pretraining via Edges — arXiv:2609.02881 — arXiv (en), https://arxiv.org/abs/2609.02881 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eccaf7d76889 (Metraipolis archive dc_eccaf7d76889, sha256 5a39bbf17453ec14a7c0b65d8c430d740b874de628ceccf6c0be1d051916fb3a)
BibTeX
@misc{dceccaf7d76889,
  title = {Graph Machine: Towards Better Pretraining via Edges — arXiv:2609.02881},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.02881},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eccaf7d76889; sha256 5a39bbf17453ec14a7c0b65d8c430d740b874de628ceccf6c0be1d051916fb3a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.02881. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We introduce the Graph Machine (GM), an architecture that maintains an $O(n)$-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves $O(n)$ complexity in its sparse layers without restricting the potentially accessible state size to $O(1)$. Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-03 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.02881 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T04:09:43.340Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-03?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).