Metraipolis › archive › dc_41ed8ca2bd9b

POSPAN: Position-Constrained Span Masking for Language Model Pre-training — arXiv:2609.16061

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.16061
License (as archived)
CC0-1.0
Archived
2026-10-15T18:32:34.733Z
SHA-256 of the archived text
bd7ffc70ef6b4194a62dd776f0e3d8ae9caf24657aab7b905506e79d18946d78
Stable URL of this page
https://metraipolis.com/documento/dc_41ed8ca2bd9b

How to cite

POSPAN: Position-Constrained Span Masking for Language Model Pre-training — arXiv:2609.16061 — arXiv (en), https://arxiv.org/abs/2609.16061 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_41ed8ca2bd9b (Metraipolis archive dc_41ed8ca2bd9b, sha256 bd7ffc70ef6b4194a62dd776f0e3d8ae9caf24657aab7b905506e79d18946d78)
BibTeX
@misc{dc41ed8ca2bd9b,
  title = {POSPAN: Position-Constrained Span Masking for Language Model Pre-training — arXiv:2609.16061},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.16061},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_41ed8ca2bd9b; sha256 bd7ffc70ef6b4194a62dd776f0e3d8ae9caf24657aab7b905506e79d18946d78},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16061. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Span-level masked language modeling (MLM) has shown to be advantageous to pre-trained language models over the original single-token MLM, as entities/phrases and their dependencies are critical to language understanding. Previous works only consider span length with some discrete distributions, while the dependencies among spans are ignored, i.e., assuming that the positions of masked spans are uniformly distributed. In this paper, we present POSPAN, a general framework to allow diverse position-constrained span masking strategies via the combination of span length distribution and position constraint distribution, which unifies all existing span-level masking methods. To verify the effectiveness of POSPAN in pre-training, we evaluate it on the datasets from several NLU benchmarks. Experimental results indicate that the position constraint is capable of enhancing span-level masking broadly, and our best POSPAN setting consistently outperforms its span-length-only counterparts and vanilla MLM. We also conduct theoretical analysis for the position constraint in masked language models to shed light on the reason why POSPAN works well, demonstrating the rationality and necessity of POSPAN.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.16061 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T18:03:09.563Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).