Metraipolis › archive › dc_ca01f24ee1cd
SciTBERT: A family of chronologically consistent language models for scientific and technological language processing — arXiv:2610.12207
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.12207
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:13:28.428Z
- SHA-256 of the archived text
0d05829b1781557cf6455fcdf4e0dd6cb63de2615016516e7936da20c84fbee6- Stable URL of this page
https://metraipolis.com/documento/dc_ca01f24ee1cd
How to cite
SciTBERT: A family of chronologically consistent language models for scientific and technological language processing — arXiv:2610.12207 — arXiv (en), https://arxiv.org/abs/2610.12207 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ca01f24ee1cd (Metraipolis archive dc_ca01f24ee1cd, sha256 0d05829b1781557cf6455fcdf4e0dd6cb63de2615016516e7936da20c84fbee6)
BibTeX
@misc{dcca01f24ee1cd,
title = {SciTBERT: A family of chronologically consistent language models for scientific and technological language processing — arXiv:2610.12207},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.12207},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ca01f24ee1cd; sha256 0d05829b1781557cf6455fcdf4e0dd6cb63de2615016516e7936da20c84fbee6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.12207. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Pre-trained transformer models are increasingly being used to study scientific and technological progress. Encoders tuned to paper or patent text outperform general-purpose models on downstream classification, regression, and proximity tasks within science and technology. However, the applicability of these models for studying time-dependent or archival properties of science, technology, and their interface is limited due to lookahead and domain biases inherent to these pre-trained models. These limitations arise from training on corpora with unconstrained chronological and text source distributions. We introduce SciTBERT: a family of chronologically consistent BERT-derived language models trained on text from scientific papers, patents, and high-quality educational web text with training data cutoff dates spanning each year between 2013 and 2025. We also post-train these models in a chronologically-consistent manner using paper and patent citations, creating SciTBERT-CI model family. We find that these models generally outperform predecessor domain-specific encoder models even when training data is limited by early year restrictions in the corpus. To further investigate the extent to which this class of models can learn representations that bridge science and technology, we introduce the PatRepEval benchmark, a suite of patent-related text embedding tasks at the science-technology interface. Performance in a variety of classification, regression, and retrieval tasks spanning papers and patents highlights the importance of aligning encoder model representations with the domain distributions of their downstream tasks, and chronologically consistent encoders can match or exceed models trained without temporal constraints. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.12207 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:47:15.393Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).