Metraipolis › archive › dc_d1bef9024972

HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning — arXiv:2611.02681

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02681
License (as archived)
CC0-1.0
Archived
2026-11-03T10:58:35.611Z
SHA-256 of the archived text
9968955af57f3692ce7b381fd99b8d4712deaec86892ed973e59d8d76549a318
Stable URL of this page
https://metraipolis.com/documento/dc_d1bef9024972

How to cite

HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning — arXiv:2611.02681 — arXiv (en), https://arxiv.org/abs/2611.02681 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d1bef9024972 (Metraipolis archive dc_d1bef9024972, sha256 9968955af57f3692ce7b381fd99b8d4712deaec86892ed973e59d8d76549a318)
BibTeX
@misc{dcd1bef9024972,
  title = {HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning — arXiv:2611.02681},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02681},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d1bef9024972; sha256 9968955af57f3692ce7b381fd99b8d4712deaec86892ed973e59d8d76549a318},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02681. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Spatial transcriptomics maps gene expression across tissue sections and, when integrated with whole slide images, jointly captures both morphological architecture and molecular activity to provide a rich, multimodal view of tissue biology. ML approaches combining spatial transcriptomics and WSIs typically learn a mapping from morphological features to molecular profiles via contrastive alignment or supervised fine-tuning of pathology foundation models. In both cases, transcriptomics serves as a training signal rather than a true input modality, leaving inference fundamentally image-driven. We propose an early-fusion vision transformer architecture in which transcript tokens are constructed at arbitrary spatial resolution, merged with image patch tokens, and jointly processed through a shared transformer stack for deep cross-modal interaction. Starting from a pretrained vision-only pathology foundation model, we train HistoTx by continuing self-supervised pretraining on paired image and spatial transcriptomics data and demonstrate that: 1) at inference with image only, performance remains on par with the baseline on most tasks while improving on tasks that inherently benefit from molecular knowledge, such as gene expression prediction; 2) when both modalities are available at inference, jointly providing image and transcript inputs outperforms either modality alone.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Genomics (q-bio.GN); Quantitative Methods (q-bio.QM).
Pagina del lavoro: https://arxiv.org/abs/2611.02681 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:49:38.667Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).