Metraipolis › archive › dc_4b04a71fa83e

Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining — arXiv:2609.07798

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.07798
License (as archived)
CC0-1.0
Archived
2026-10-16T12:45:47.174Z
SHA-256 of the archived text
7dbaaf770077628c67f69b73c81cf834c2bb3f8343ad940703e8255660b2bed9
Stable URL of this page
https://metraipolis.com/documento/dc_4b04a71fa83e

How to cite

Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining — arXiv:2609.07798 — arXiv (en), https://arxiv.org/abs/2609.07798 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4b04a71fa83e (Metraipolis archive dc_4b04a71fa83e, sha256 7dbaaf770077628c67f69b73c81cf834c2bb3f8343ad940703e8255660b2bed9)
BibTeX
@misc{dc4b04a71fa83e,
  title = {Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining — arXiv:2609.07798},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.07798},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4b04a71fa83e; sha256 7dbaaf770077628c67f69b73c81cf834c2bb3f8343ad940703e8255660b2bed9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.07798. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Natural Language Processing (NLP) in the climate domain requires models to process heterogeneous text sources, including scientific literature, policy disclosures, and synthetic reports. However, how to effectively combine diverse domain corpora during continued pretraining (CPT) remains underexplored. We introduce Climate-ModernBERT, a family of climate-adapted encoder models obtained through continued pretraining of ModernBERT-Base on three climate corpora: academic climate text, climate-filtered web data, and synthetic climate documents. We systematically compare joint continued pretraining on corpus mixtures with parameter-space merging of independently specialized checkpoints. Across nine climate NLP benchmarks, our best model achieves 76.3 average F_1, improving significantly over a vanilla ModernBERT baseline by 2.8 points. Within the climate NLP setting, the results show that academic climate corpora provide the strongest adaptation signal among the evaluated sources, while parameter-space merging improves over joint multi-source training and better preserves complementary information from heterogeneous climate corpora. We release all Climate-ModernBERT variants and training checkpoints to support future research in climate NLP and domain-adaptive pretraining.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.07798 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T12:30:25.145Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).