Metraipolis › archive › dc_15fd52291bd0

PairPref: When Should Memory Guide the Answer? A Benchmark for Contextual Preference Use — arXiv:2609.34526

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.34526
License (as archived)
CC0-1.0
Archived
2026-10-07T20:57:51.677Z
SHA-256 of the archived text
950d706652612c1e66f892e69069098009c2ec9fa877dd8661ffe04810a6288c
Stable URL of this page
https://metraipolis.com/documento/dc_15fd52291bd0

How to cite

PairPref: When Should Memory Guide the Answer? A Benchmark for Contextual Preference Use — arXiv:2609.34526 — arXiv (en), https://arxiv.org/abs/2609.34526 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_15fd52291bd0 (Metraipolis archive dc_15fd52291bd0, sha256 950d706652612c1e66f892e69069098009c2ec9fa877dd8661ffe04810a6288c)
BibTeX
@misc{dc15fd52291bd0,
  title = {PairPref: When Should Memory Guide the Answer? A Benchmark for Contextual Preference Use — arXiv:2609.34526},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34526},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_15fd52291bd0; sha256 950d706652612c1e66f892e69069098009c2ec9fa877dd8661ffe04810a6288c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34526. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Memory-augmented assistants use retrieved preferences to guide their responses. A small change in the situation can change whether a preference is appropriate while barely affecting its retrieval similarity. Memory benchmarks typically test whether systems store and retrieve preferences, with less attention to when those preferences should apply. We introduce PairPref, a benchmark of contextual preference use. Each pair changes only the situation, keeping the preference, request, and four candidate replies fixed. The preference remains valid in both situations. In the selection track, models must choose the reply that applies the preference only where appropriate. In the free-generation track, they must decide when to apply it without seeing candidate replies. Both tracks use the same 1,227 pairs across 45 preferences and eight situation categories. We evaluate eight models, most of which achieve selection scores ($\Delta$) of 51 to 65 points. In free generation, however, both responses are appropriate for their respective situations in only 3.6\% to 18.3\% of pairs. Models continue to apply the preference in both situations even with fewer retrieved memories, alternative presentation formats, and a stricter prompt. These results show that models still struggle to judge when user preferences apply and respond accordingly.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.34526 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.572Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).