Metraipolis › archive › dc_6325fb16e617

Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items — arXiv:2610.40241

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.40241
License (as archived)
CC0-1.0
Archived
2026-10-31T06:17:06.673Z
SHA-256 of the archived text
2ff8b4d508e919d8f5dc660b8cc5b4e83e4a52980c760bf08df9cf083895763e
Stable URL of this page
https://metraipolis.com/documento/dc_6325fb16e617

How to cite

Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items — arXiv:2610.40241 — arXiv (en), https://arxiv.org/abs/2610.40241 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6325fb16e617 (Metraipolis archive dc_6325fb16e617, sha256 2ff8b4d508e919d8f5dc660b8cc5b4e83e4a52980c760bf08df9cf083895763e)
BibTeX
@misc{dc6325fb16e617,
  title = {Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items — arXiv:2610.40241},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40241},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6325fb16e617; sha256 2ff8b4d508e919d8f5dc660b8cc5b4e83e4a52980c760bf08df9cf083895763e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40241. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

To save cost, practitioners often ask a language model to grade many independent items in one prompt. This assumes that verdicts are separable: that the verdict on an item does not depend on the other items in the prompt or on its position among them. Across three judgement tasks (harm in social-media comments, short-answer grading and review star ratings) and five open-weight models with 4-9B parameters, assigning the same 2,000 items to different random batches of 25 changes 11-22% of decisions on harm and grading, 6.5-13.3 points more than renaming the item keys or rewording the instruction, while AUROC against human labels differs from single-item grading only by -0.033 to +0.010; for four models with 14-32B parameters, 6-15% of decisions change, 4.2-9.1 points more. We show that flip rates decompose exactly into components of order, slot and membership, and that under a threshold model they grow with the square root of the variance that the context contributes, so that effects that are small for scores are large for decisions. Most of the latent dissimilarity between partitions comes from the order of the neighbours and the slot. The main effect of which items share the prompt, the part that voting over permutations of a batch cannot remove, accounts for 0.11-0.22 of it, and majority votes over nine permutations still leave 52-60% of the flips. Batched verdicts also differ systematically from single-item verdicts: even the majority of fifteen random batches disagrees with them on 8.4-17.8% of decisions. Editing a single generation shows that both the model's own earlier verdicts and the neighbours' texts carry the effect. On the easy task, star ratings, only 2.9-3.9% of decisions change. We recommend grading ambiguous items one per prompt, recovering most of the cost with a cached prefix, and reporting agreement with single-item grading whenever items are batched.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.40241 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:23.371Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).