Metraipolis › archive › dc_fb95e98ba06d
Invariant Pretraining for Robust Code Representations — arXiv:2608.15412
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.15412
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T18:03:27.425Z
- SHA-256 of the archived text
0ecd692357ba0868ddcb894bf112fdb3d1b398bd749599e6ea1360da1f0ca174- Stable URL of this page
https://metraipolis.com/documento/dc_fb95e98ba06d
How to cite
Invariant Pretraining for Robust Code Representations — arXiv:2608.15412 — arXiv (en), https://arxiv.org/abs/2608.15412 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fb95e98ba06d (Metraipolis archive dc_fb95e98ba06d, sha256 0ecd692357ba0868ddcb894bf112fdb3d1b398bd749599e6ea1360da1f0ca174)
BibTeX
@misc{dcfb95e98ba06d,
title = {Invariant Pretraining for Robust Code Representations — arXiv:2608.15412},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.15412},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fb95e98ba06d; sha256 0ecd692357ba0868ddcb894bf112fdb3d1b398bd749599e6ea1360da1f0ca174},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.15412. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Encoder-based code representation models remain widely deployed for discriminative tasks such as clone detection and code classification, where their small size and low inference cost are decisive. Their robustness, however, is fragile: under invariant programs, semantically equivalent code written in different syntactic forms, learned representations degrade substantially even though program behavior is unchanged. We present an empirical study of this robustness gap across four encoder baselines, two downstream tasks, and four datasets, together with a minimal code-only continued pretraining recipe that closes much of it. Our method, invariant pretraining (InvPT), applies semantic-preserving transformations to the corpus and combines masked language modeling with multi-positive supervised contrastive learning that treats all augmentations of the same source function as positives, mixing self-contrast pairs (same code, different masks) with invariant-contrast pairs (transformed code) for positives of varying difficulty. Unlike prior contrastive code encoders, InvPT does not require paired natural-language data. Across our evaluation, InvPT improves robustness on transformed test sets by up to 11 percentage points on clone detection and 19 on code classification while matching or improving standard accuracy, and our ablations isolate multi-positive invariant contrast as the main source of the gains. Our aim is not a new objective but a careful measurement of where encoder robustness breaks and how far a simple, code-only recipe can recover it. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2608.15412 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T16:29:14.016Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).