Metraipolis › archive › dc_5f95c8af5fe2

Traversing the solution space of neural networks with Hessian Null Space Continuation — arXiv:2609.38081

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.38081
License (as archived)
CC0-1.0
Archived
2026-10-07T17:32:02.811Z
SHA-256 of the archived text
810cab480c6da7f17d08f1eb0691db5be734ee25236278644c4b372ef7955ac3
Stable URL of this page
https://metraipolis.com/documento/dc_5f95c8af5fe2

How to cite

Traversing the solution space of neural networks with Hessian Null Space Continuation — arXiv:2609.38081 — arXiv (en), https://arxiv.org/abs/2609.38081 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5f95c8af5fe2 (Metraipolis archive dc_5f95c8af5fe2, sha256 810cab480c6da7f17d08f1eb0691db5be734ee25236278644c4b372ef7955ac3)
BibTeX
@misc{dc5f95c8af5fe2,
  title = {Traversing the solution space of neural networks with Hessian Null Space Continuation — arXiv:2609.38081},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.38081},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5f95c8af5fe2; sha256 810cab480c6da7f17d08f1eb0691db5be734ee25236278644c4b372ef7955ac3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38081. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On a single task, deep networks can learn many solutions, depending on their optimizer, training data, architecture, and hyperparameters. Many of these solutions are mode-connected: rather than isolated points in weight space, they are connected by low-loss regions. Yet how their internal computation varies within these regions is unknown. A parallel line of work has identified the degeneracy of neural representations: many networks reach similar training loss with distinct internal structures. However, it is unclear how these solutions are related in weight space. We unify these subfields and show for the first time that many different internal mechanisms exist within a local mode-connected region in weight space. To do so, we introduce Hessian Null Space Continuation (HNC), a scalable method that uses local curvature to traverse regions of weight space that preserve network function, and can be steered toward solutions with specified properties. In RNNs trained on a memory task, HNC reaches drastically different representations and dynamics with maintained behavior. In ImageNet-trained Vision Transformers, HNC finds representations that differ more from the original network than any independently trained model with a different architecture or objective. In reinforcement-learning agents, HNC uncovers a distinct navigation strategy at comparable return and exposes reward hacking in an AI Safety Gridworld. Finally, HNC measures the local geometry of the solution set, showing how model size and task complexity shape its dimension and functional sensitivity. Our results show that a surprisingly large amount of representational diversity exists near a single trained solution, unseen by standard gradient-based optimization. HNC identifies and quantifies this diversity, opening new possibilities for mechanistic understanding of solution spaces and for model merging, editing, and fine-tuning.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Neurons and Cognition (q-bio.NC); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.38081 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:59:48.418Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).