Metraipolis › archive › dc_f997ac4dfe61
Distilled Rapid Embedding Transfer (DRET): Parameter-Efficient Biomedical Domain Adaptation via Priority-Based Embedding Transfer — arXiv:2609.02898
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.02898
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T01:37:18.034Z
- SHA-256 of the archived text
03e30c7b77d535c06a7b1d790e6c0f5ad4001f162703681cd87c260a0dcf6da1- Stable URL of this page
https://metraipolis.com/documento/dc_f997ac4dfe61
How to cite
Distilled Rapid Embedding Transfer (DRET): Parameter-Efficient Biomedical Domain Adaptation via Priority-Based Embedding Transfer — arXiv:2609.02898 — arXiv (en), https://arxiv.org/abs/2609.02898 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f997ac4dfe61 (Metraipolis archive dc_f997ac4dfe61, sha256 03e30c7b77d535c06a7b1d790e6c0f5ad4001f162703681cd87c260a0dcf6da1)
BibTeX
@misc{dcf997ac4dfe61,
title = {Distilled Rapid Embedding Transfer (DRET): Parameter-Efficient Biomedical Domain Adaptation via Priority-Based Embedding Transfer — arXiv:2609.02898},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.02898},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f997ac4dfe61; sha256 03e30c7b77d535c06a7b1d790e6c0f5ad4001f162703681cd87c260a0dcf6da1},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.02898. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large domain-specific language models such as BioBERT and ClinicalBERT achieve strong performance on biomedical NLP tasks, but their computational demands make them impractical for many real-world deployments. General-purpose, parameter-efficient models such as DistilBERT are lightweight yet lack the domain knowledge required for specialized tasks such as PICO (Population, Intervention, Comparison, Outcome) classification. We introduce Distilled Rapid Embedding Transfer (DRET), a knowledge-transfer paradigm that injects biomedical domain knowledge from large specialized models into a smaller general-purpose model without retraining on the original specialized corpora. DRET is developed as an iterative family of strategies: a unified tokenizer-merge strategy (DRET 1.x), hybrid embedding averaging (DRET 2.0), and a priority-based embedding-transfer mechanism (DRET 3.x) that hierarchically selects embeddings from the most authoritative source models, further combined with embedding-layer freezing, differential learning rates, label propagation, and imbalance-aware loss functions (DRET 4.x). We evaluate DRET on token-level PICO classification using the EBM-NLP corpus under severe class imbalance, across a twelve-metric battery. DRET-enhanced DistilBERT (66M parameters) attains balanced accuracy, recall, and ROC-AUC competitive with, and on several class-wise metrics exceeding, models an order of magnitude larger, while retaining DistilBERT's efficiency. We further show that transfer occurs at the embedding level through cosine-similarity, semantic-shift, and t-SNE analyses. DRET offers a scalable, resource-efficient route to near-domain-expert performance for biomedical text mining, with direct application to automated systematic literature reviews and clinical decision support. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.02898 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T01:11:43.487Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).