Metraipolis › archive › dc_44e604d12930

Every Clause Counts: Instruction-Following Reranking by Multiplying Small, Answerable Questions — arXiv:2611.03179

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.03179
License (as archived)
CC0-1.0
Archived
2026-11-03T11:11:56.058Z
SHA-256 of the archived text
33a78fd0e6e60e85d0b9cfabf8164ec2b23c39581035ea72ff2d24ed27ccf4e3
Stable URL of this page
https://metraipolis.com/documento/dc_44e604d12930

How to cite

Every Clause Counts: Instruction-Following Reranking by Multiplying Small, Answerable Questions — arXiv:2611.03179 — arXiv (en), https://arxiv.org/abs/2611.03179 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_44e604d12930 (Metraipolis archive dc_44e604d12930, sha256 33a78fd0e6e60e85d0b9cfabf8164ec2b23c39581035ea72ff2d24ed27ccf4e3)
BibTeX
@misc{dc44e604d12930,
  title = {Every Clause Counts: Instruction-Following Reranking by Multiplying Small, Answerable Questions — arXiv:2611.03179},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03179},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_44e604d12930; sha256 33a78fd0e6e60e85d0b9cfabf8164ec2b23c39581035ea72ff2d24ed27ccf4e3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03179. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

People often attach instructions to their search queries, asking for pages written for beginners, documents in a certain language or official sources only. Most rerankers were trained to judge topical relevance and treat such instructions as a few extra words. On the InfoSearch benchmark, even GPT-4o used as a list-wise reranker reaches an average WISE (Weighted Instruction Sensitivity Evaluation) of only 33.5. We present DMReranker (Decision-Model Reranker), a zero-shot reranker that splits an instructed query into a topic and a short list of typed clauses. A small judge model answers one narrow question per clause, and the answer probabilities are multiplied into a single score. Nothing is trained, and the configuration was frozen before the test runs. Reranking the same top-100 candidates as the published baselines, DMReranker reaches a macro WISE of 50.9 over all six InfoSearch dimensions while keeping original-mode nDCG@10 at 82.9. It trails GPT-4o on the stricter SICR (Strict Instruction Compliance Ratio) metric, 19.2 against 32.1. We show that this gap follows from how a product of probabilities behaves when SICR compares raw scores across runs. Computed from ranks, our SICR is 67.0. On the same judge, splitting the instruction raises WISE from 32.9 to 47.6 over a single combined question, while raw-score SICR falls. Scoring all six dimensions cost about $10 at list price.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Information Retrieval (cs.IR) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.03179 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:52.426Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).