Metraipolis › archive › dc_bbfd7893d319

Instruction Quality Matters: Refining Instructions for Effective Preference Learning — arXiv:2608.26779

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.26779
License (as archived): CC0-1.0
Archived: 2026-10-17T15:19:57.362Z
SHA-256 of the archived text: 8a18bea35ff5b2734df0e11e6a27f233fd2203d92e7ac5e65fd3fa47f38a28a6
Stable URL of this page: https://metraipolis.com/documento/dc_bbfd7893d319

How to cite

Instruction Quality Matters: Refining Instructions for Effective Preference Learning — arXiv:2608.26779 — arXiv (en), https://arxiv.org/abs/2608.26779 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bbfd7893d319 (Metraipolis archive dc_bbfd7893d319, sha256 8a18bea35ff5b2734df0e11e6a27f233fd2203d92e7ac5e65fd3fa47f38a28a6)

BibTeX

@misc{dcbbfd7893d319,
  title = {Instruction Quality Matters: Refining Instructions for Effective Preference Learning — arXiv:2608.26779},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.26779},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bbfd7893d319; sha256 8a18bea35ff5b2734df0e11e6a27f233fd2203d92e7ac5e65fd3fa47f38a28a6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26779. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Preference learning optimizes models using response pairs, yet the informativeness of these pairs is fundamentally shaped by the instructions from which they are generated. We identify instruction quality as a hidden bottleneck in preference learning: low-quality or ambiguous instructions restrict the response-quality distribution, limiting strong chosen responses and weakening preference signals. Through Best- and Worst-of-N analyses, we show that instruction quality constrains both the ceiling and floor of sampled response quality. Motivated by this observation, we introduce an instruction-refinement pipeline that selects weak instructions using reward signals and revises them with rubric-guided LLM feedback, improving preference data without discarding examples. Across offline and online preference learning settings, experiments on multiple models and benchmarks show broad alignment improvements over original data and alternative data-improvement strategies. Further analyses indicate that instruction refinement raises achievable response quality and complements response-centric preference data curation. Overall, instruction quality emerges as a key factor governing how informative preference signals are formed for LLM alignment. Code is available at: this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.26779 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T15:15:18.073Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).