Metraipolis › archive › dc_b66058808abc
Language Models Can Benefit from Domain-Specific Inductive Biases — arXiv:2611.03756
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.03756
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T12:39:19.417Z
- SHA-256 of the archived text
4f9cf808cfb00bde66a5ce2e25d5127b002582a647f88eb620634be05ea278e4- Stable URL of this page
https://metraipolis.com/documento/dc_b66058808abc
How to cite
Language Models Can Benefit from Domain-Specific Inductive Biases — arXiv:2611.03756 — arXiv (en), https://arxiv.org/abs/2611.03756 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b66058808abc (Metraipolis archive dc_b66058808abc, sha256 4f9cf808cfb00bde66a5ce2e25d5127b002582a647f88eb620634be05ea278e4)
BibTeX
@misc{dcb66058808abc,
title = {Language Models Can Benefit from Domain-Specific Inductive Biases — arXiv:2611.03756},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.03756},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b66058808abc; sha256 4f9cf808cfb00bde66a5ce2e25d5127b002582a647f88eb620634be05ea278e4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03756. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Language models are typically trained with a single choice of architecture on diverse domains and data sources. Modern transformers have been extensively refined for such general-purpose learning, but are they equally effective across different subsets of the pretraining data? We investigate this question by training small language models from scratch on representative subsets of pretraining data such as web text, code, and mathematics. We introduce variants of transformers in which standard activation functions are replaced with learned splines tuned on specific datasets, providing a simple way to modify their inductive biases. Using a mix-and-match of architectures and datasets, we evaluate cross-domain transfer, i.e. how well does an architecture tuned for one domain perform on data from another? We find that the tuned transformer variants improve training efficiency or generalization on all seven of our tested datasets, but the gains are often domain-specific. For example, architectures tuned on a natural language dataset such as FineWeb perform well on TinyStories, whereas those tuned on mathematics perform poorly on natural language. We identify several mechanisms underlying these improvements, including wider loss basins and more abundant "lottery tickets". Our results show that different subsets of pretraining data can benefit from different learning mechanisms, highlighting a limitation of the prevailing trend towards a small set of general-purpose architectures and optimizers. This points to an opportunity to improve data efficiency by developing domain-specific learning strategies that retain the ability to train general-purpose models. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.03756 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T12:29:47.244Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).