Metraipolis › archive › dc_5d643b4d7c19

DimPO: Dimensionality Reduction for Attention using Preference Optimization — arXiv:2609.32579

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.32579
License (as archived): CC0-1.0
Archived: 2026-10-07T22:36:41.927Z
SHA-256 of the archived text: f1837b9191fb016a4071a2304ef4dd086ef1f2e368c11de946b9c4c1d7be82e9
Stable URL of this page: https://metraipolis.com/documento/dc_5d643b4d7c19

How to cite

DimPO: Dimensionality Reduction for Attention using Preference Optimization — arXiv:2609.32579 — arXiv (en), https://arxiv.org/abs/2609.32579 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5d643b4d7c19 (Metraipolis archive dc_5d643b4d7c19, sha256 f1837b9191fb016a4071a2304ef4dd086ef1f2e368c11de946b9c4c1d7be82e9)

BibTeX

@misc{dc5d643b4d7c19,
  title = {DimPO: Dimensionality Reduction for Attention using Preference Optimization — arXiv:2609.32579},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32579},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5d643b4d7c19; sha256 f1837b9191fb016a4071a2304ef4dd086ef1f2e368c11de946b9c4c1d7be82e9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32579. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A linear projection can reduce the dimension of query and key vectors without updating the pretrained model, but it remains unclear which training objective best preserves model behavior. We ask whether preferences over keys and attention mass on the highest-weighted keys provide a better signal than matching the full attention distribution with KL divergence, especially in long-context settings. We introduce DimPO, which combines listwise preference optimization with a lightweight top-k cross-entropy term for head-fidelity. DimPO is trained offline from the attention patterns of a frozen language model, with one map per layer, shared by the query and the keys and trained separately from the other layers. Across LLaMA3.2-3B, LLaMA3.1-8B, Qwen2.5-7B, and Qwen3-4B Instruct models, pairwise preference objectives outperform the triplet baseline and retain 98% of the original score on short-context tasks when projecting to half the dimension on the last 40% of the layers. With more projected layers or on long-context RULER, they degrade rapidly. In contrast, KL and DimPO, which use every key during training, retain about 95% of the original RULER 4k score on the 8B model when projecting up to 50% of the layers. KL-based projections remain closer to the original attention distribution and attention output, yet DimPO achieves better downstream performance. Beyond 50% of projected layers, DimPO increasingly outperforms KL on tasks including SQuAD, common-word extraction, frequent-word extraction, and variable tracking. These results suggest that under dimensionality reduction, preserving the ordering and concentration of task-relevant attention can matter more than reproducing the full attention distribution.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.32579 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:27.722Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).