Metraipolis › archive › dc_0dc074a086cb

Auditing Pairwise Equivalence Judgments: Self-Critique Effects and Diversity Measurement in Multi-Agent Hypothesis Generation — arXiv:2610.04133

Source
arXiv
Original
https://arxiv.org/abs/2610.04133
License (as archived)
CC0-1.0
Archived
2026-10-06T05:24:19.588Z
SHA-256 of the archived text
67b6c6dbd81926ee03adcafb3a71d3f1961a9afb8d3e457ce29974941cb8a0b0
Stable URL of this page
https://metraipolis.com/documento/dc_0dc074a086cb

How to cite

Auditing Pairwise Equivalence Judgments: Self-Critique Effects and Diversity Measurement in Multi-Agent Hypothesis Generation — arXiv:2610.04133 — arXiv, https://arxiv.org/abs/2610.04133 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0dc074a086cb (Metraipolis archive dc_0dc074a086cb, sha256 67b6c6dbd81926ee03adcafb3a71d3f1961a9afb8d3e457ce29974941cb8a0b0)
BibTeX
@misc{dc0dc074a086cb,
  title = {Auditing Pairwise Equivalence Judgments: Self-Critique Effects and Diversity Measurement in Multi-Agent Hypothesis Generation — arXiv:2610.04133},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.04133},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0dc074a086cb; sha256 67b6c6dbd81926ee03adcafb3a71d3f1961a9afb8d3e457ce29974941cb8a0b0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04133. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multi-agent systems built on large language models (LLMs) are increasingly applied to scientific discovery and hypothesis generation. Both the effect of refinement and the diversity of the delivered set are hard to interpret before experimental ground truth exists, and both are typically reported by deciding whether pairs of generated hypotheses describe the same underlying mechanism. We study two evaluation questions that rest on this pairwise equivalence judgment: (1) how much self-critique changes delivered hypotheses beyond run-to-run variability, and (2) how the equivalence rule used to group hypotheses affects measured diversity. Across four proprietary instances, we hold opening hypotheses fixed, rerun the downstream workflow with 0, 1, and 5 critique rounds, and score matched hypothesis pairs with an LLM-as-a-judge. Relative to matched same-depth reruns, moving from 0 to 1 round produces 34.5 percentage points (pp) of additional mechanism-level divergence, whereas 1 to 5 rounds adds 1.3 pp. We then compare three equivalence rules: term frequency--inverse document frequency (TF--IDF) similarity, dense embeddings, and the same LLM-as-a-judge. We construct controlled hypothesis pairs that either preserve the causal explanation through wording or biological-terminology changes, or replace one component of the causal chain while holding the rest fixed. All three rules are invariant to meaning-preserving edits, but when the initiating event is replaced, the LLM-as-a-judge identifies 83% of valid pairs as different mechanisms, versus 0% for TF--IDF and 8% for embeddings; varying only the rubric that defines same mechanism moves this figure from 38% to 96%. Together, these results show that pairwise equivalence judgments are a measurement choice: how mechanism equivalence is defined affects both the estimated effect of self-critique and the measured diversity of generated hypotheses.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.04133 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:05.431Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).