Metraipolis › archive › dc_8ef898dbbd74

Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay — arXiv:2610.05564

Source
arXiv
Original
https://arxiv.org/abs/2610.05564
License (as archived)
CC0-1.0
Archived
2026-10-06T05:28:22.490Z
SHA-256 of the archived text
44cc80c87ed1e75d7a27b218cc77e8587e3e6e6503db8284a71946d0c8cb742e
Stable URL of this page
https://metraipolis.com/documento/dc_8ef898dbbd74

How to cite

Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay — arXiv:2610.05564 — arXiv, https://arxiv.org/abs/2610.05564 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8ef898dbbd74 (Metraipolis archive dc_8ef898dbbd74, sha256 44cc80c87ed1e75d7a27b218cc77e8587e3e6e6503db8284a71946d0c8cb742e)
BibTeX
@misc{dc8ef898dbbd74,
  title = {Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay — arXiv:2610.05564},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05564},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8ef898dbbd74; sha256 44cc80c87ed1e75d7a27b218cc77e8587e3e6e6503db8284a71946d0c8cb742e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05564. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Text embedding models trained with contrastive learning learn to follow task instructions from instruction-paired data, while instruction-tuned LLMs already know how to follow them. We show that this instruction-following ability can carry over from an LLM to a Transformer-based embedder without any training. We propose Attention Relay, which passes the attention weights an LLM produces to the embedder's own attention. Across six instruction-tuned LLMs from the Qwen3, Llama 3.1 and OLMo 3 families and ten widely used embedding models that differ in tokenizer, size and pooling type, Attention Relay makes nearly every combination instruction-aware. Experiments that break the method down into its parts show that the LLM's attention weights track the instruction in its later layers and come largely from instruction tuning. They also show that relaying these weights selects which content in the text matters: it makes the aspect of the text that the instruction asks about dominant in the embedding, or restores that aspect where averaging had diluted it.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.05564 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:20.336Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).