Metraipolis › archive › dc_d3c08eeea3b8

TractoGraphVLM: A Unified Vision-Language Framework for White Matter Tractography — arXiv:2608.18166

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.18166
License (as archived)
CC0-1.0
Archived
2026-10-18T14:06:35.879Z
SHA-256 of the archived text
ddb390d3e83b46740b269a4d2dac256bdaef9dfe08b4fc7ff62aa39ea0f3a5ba
Stable URL of this page
https://metraipolis.com/documento/dc_d3c08eeea3b8

How to cite

TractoGraphVLM: A Unified Vision-Language Framework for White Matter Tractography — arXiv:2608.18166 — arXiv (en), https://arxiv.org/abs/2608.18166 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d3c08eeea3b8 (Metraipolis archive dc_d3c08eeea3b8, sha256 ddb390d3e83b46740b269a4d2dac256bdaef9dfe08b4fc7ff62aa39ea0f3a5ba)
BibTeX
@misc{dcd3c08eeea3b8,
  title = {TractoGraphVLM: A Unified Vision-Language Framework for White Matter Tractography — arXiv:2608.18166},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.18166},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d3c08eeea3b8; sha256 ddb390d3e83b46740b269a4d2dac256bdaef9dfe08b4fc7ff62aa39ea0f3a5ba},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18166. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vision language models have transformed 2D medical imaging, yet extending them to 3D white matter tractography remains challenging due to the complex topology of fiber bundles. We introduce TractoGraphVLM, a unified framework for four tasks, bundle classification, text-to-tract retrieval, anatomical captioning, and visual question answering, built on a shared GPS architecture, training procedure, and read-out design. Fiber bundles are represented as streamline graphs whose nodes encode 3D position and tangent orientation. A General, Powerful, Scalable (GPS) graph transformer produces bundle embeddings aligned with a frozen BiomedBERT text encoder via contrastive learning, while a BioGPT decoder with visual prefix tokens generates captions and answers. A single shared encoder and decoder is trained jointly across all four tasks and evaluated from one checkpoint. Trained on HCP Young Adult subjects, TractoGraphVLM achieves 91.8% bundle classification accuracy, 84.7% retrieval R@1, BLEU-4=20.1, ROUGE-L=66.8, and 66.4% VQA accuracy on a held-out test set. The same checkpoints transfer zero-shot to HCP Aging subjects, with a modest drop on discriminative tasks and a larger drop on generative tasks, showing robustness to age and acquisition shift. Language supervision yields richer representations than label-only training, recovering structure like hemisphere and fiber family, carried by captions but never given as a label. Swapping only the visual encoder, graphs preserving fiber orientation outperform volumetric baselines, with GPS giving the best balance. Generative metrics measure consistency with a structured knowledge base rather than independent clinical text; even so, TractoGraphVLM shows that classifying, retrieving, describing, and answering questions about a white matter bundle can be served by one jointly trained model that learns transferable neuroanatomy from language alone.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-08-20 in cs.CV).
Categorie arXiv: Image and Video Processing (eess.IV) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.18166 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T14:00:22.599Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).