Metraipolis › archive › dc_8ef898dbbd74
Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay — arXiv:2610.05564
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.05564
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:28:22.490Z
- SHA-256 of the archived text
44cc80c87ed1e75d7a27b218cc77e8587e3e6e6503db8284a71946d0c8cb742e- Stable URL of this page
https://metraipolis.com/documento/dc_8ef898dbbd74
How to cite
Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay — arXiv:2610.05564 — arXiv, https://arxiv.org/abs/2610.05564 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8ef898dbbd74 (Metraipolis archive dc_8ef898dbbd74, sha256 44cc80c87ed1e75d7a27b218cc77e8587e3e6e6503db8284a71946d0c8cb742e)
BibTeX
@misc{dc8ef898dbbd74,
title = {Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay — arXiv:2610.05564},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.05564},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8ef898dbbd74; sha256 44cc80c87ed1e75d7a27b218cc77e8587e3e6e6503db8284a71946d0c8cb742e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05564. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Text embedding models trained with contrastive learning learn to follow task instructions from instruction-paired data, while instruction-tuned LLMs already know how to follow them. We show that this instruction-following ability can carry over from an LLM to a Transformer-based embedder without any training. We propose Attention Relay, which passes the attention weights an LLM produces to the embedder's own attention. Across six instruction-tuned LLMs from the Qwen3, Llama 3.1 and OLMo 3 families and ten widely used embedding models that differ in tokenizer, size and pooling type, Attention Relay makes nearly every combination instruction-aware. Experiments that break the method down into its parts show that the LLM's attention weights track the instruction in its later layers and come largely from instruction tuning. They also show that relaying these weights selects which content in the text matters: it makes the aspect of the text that the instruction asks about dominant in the embedding, or restores that aspect where averaging had diluted it. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.05564 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T05:24:20.336Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).