Metraipolis › archive › dc_5588f38737fc

PoolBench: A Benchmark for Pooling Strategies in Concept Representation Evaluation for Decoder-Only LLMs — arXiv:2608.05162

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.05162
License (as archived)
CC0-1.0
Archived
2026-10-25T21:27:01.579Z
SHA-256 of the archived text
6c5e79aac76bba801230f323f2c7e65a6d34b25d79272f5e979a32b3f939d081
Stable URL of this page
https://metraipolis.com/documento/dc_5588f38737fc

How to cite

PoolBench: A Benchmark for Pooling Strategies in Concept Representation Evaluation for Decoder-Only LLMs — arXiv:2608.05162 — arXiv (en), https://arxiv.org/abs/2608.05162 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5588f38737fc (Metraipolis archive dc_5588f38737fc, sha256 6c5e79aac76bba801230f323f2c7e65a6d34b25d79272f5e979a32b3f939d081)
BibTeX
@misc{dc5588f38737fc,
  title = {PoolBench: A Benchmark for Pooling Strategies in Concept Representation Evaluation for Decoder-Only LLMs — arXiv:2608.05162},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.05162},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5588f38737fc; sha256 6c5e79aac76bba801230f323f2c7e65a6d34b25d79272f5e979a32b3f939d081},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.05162. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Pooling is a consequential but under-examined design choice in decoder-only concept representation work: practitioners must collapse token-level hidden states into a passage-level vector, yet no shared protocol exists for comparing this choice across concepts, models, and tasks. Reported gains are confounded by simultaneous changes in dataset, layer, construction method, and pooling rule, making principled decisions impossible. We introduce PoolBench, a benchmark that isolates pooling as the experimental variable under a fixed evaluation protocol. PoolBench covers 17 concepts, 19 pooling strategies, and 3 open-weight decoder-only models (Llama-3.1-8B, Gemma-2-9B, Mistral-7B), evaluated on a single audited corpus of 37,693 real-text passages. The primary axis is linear separability (D1/AUROC); steered concept prevalence (D2/SCP) and output-level disentanglement (D3) serve as diagnostic axes. The primary finding is decisive: W4_hierarchical reaches a cross-model mean AUROC of 0.7799, while the widely adopted P1_last_token baseline reaches only 0.7640 and is statistically significantly worse (Friedman+Nemenyi, p = 2.0e-36; 77 significant pairs among 18 effective strategies). Rankings are stable across layers (rho = 0.961--0.990). A key negative result: strong detection does not imply strong steering -- D2 and D3 are substantially weaker than D1 for most concepts, indicating a fundamental representational limit rather than a pooling failure. On mid-difficulty concepts, W4_hierarchical outperforms P1_last_token by 0.042--0.113 AUROC; construction method choice (DiffMean vs. REPE) has a larger effect (delta AUROC 0.15) than pooling (delta AUROC 0.016), establishing the correct practical hierarchy. We release the corpus, pre-extracted activations, scorer models, steering vectors, and evaluation code as a reusable protocol for pooling research.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-07 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.05162 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T21:24:38.170Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).