Metraipolis › archive › dc_0eeabd47bc71

ConvPE: Convolutional Position Embedding Improves Transformers — arXiv:2611.07749

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.07749
License (as archived): CC0-1.0
Archived: 2026-11-06T14:50:42.199Z
SHA-256 of the archived text: 503764a39413c7d65d815124cd7cd828e75c842945fe653a5f537727f3799d55
Stable URL of this page: https://metraipolis.com/documento/dc_0eeabd47bc71

How to cite

ConvPE: Convolutional Position Embedding Improves Transformers — arXiv:2611.07749 — arXiv (en), https://arxiv.org/abs/2611.07749 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0eeabd47bc71 (Metraipolis archive dc_0eeabd47bc71, sha256 503764a39413c7d65d815124cd7cd828e75c842945fe653a5f537727f3799d55)

BibTeX

@misc{dc0eeabd47bc71,
  title = {ConvPE: Convolutional Position Embedding Improves Transformers — arXiv:2611.07749},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07749},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0eeabd47bc71; sha256 503764a39413c7d65d815124cd7cd828e75c842945fe653a5f537727f3799d55},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07749. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

ConvPE (Convolutional Position Embedding) improves both the performance and efficiency of Transformer models in language modeling, long-context extension, and vision. It injects a windowed, content-dependent bias into attention scores through a lightweight causal convolution over a low-dimensional projection of the hidden states. As a lightweight plug-in, ConvPE is compatible with RoPE, attention masks, and KV concatenation. It also supports high-dimensional position embeddings via high-dimensional convolutions. In a reproduction, ConvPE outperforms five 1D baselines and eight 2D baselines. The code is publicly available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.07749 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:21:42.633Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).