Metraipolis › archive › dc_4993bf76fd6f

Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models — arXiv:2609.12475

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.12475
License (as archived)
CC0-1.0
Archived
2026-10-16T03:08:22.751Z
SHA-256 of the archived text
28175dafdf705076fa8250011d57ca33bfa4091d9eaf6348f435ee27378ed1fc
Stable URL of this page
https://metraipolis.com/documento/dc_4993bf76fd6f

How to cite

Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models — arXiv:2609.12475 — arXiv (en), https://arxiv.org/abs/2609.12475 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4993bf76fd6f (Metraipolis archive dc_4993bf76fd6f, sha256 28175dafdf705076fa8250011d57ca33bfa4091d9eaf6348f435ee27378ed1fc)
BibTeX
@misc{dc4993bf76fd6f,
  title = {Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models — arXiv:2609.12475},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.12475},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4993bf76fd6f; sha256 28175dafdf705076fa8250011d57ca33bfa4091d9eaf6348f435ee27378ed1fc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12475. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Comprehensive benchmark suites are essential for improving large language models (LLMs), but many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Although recent benchmark compression methods (BCMs) can mitigate this cost, many strong BCMs rely on large collections of per-sample evaluation results from numerous LLMs to identify representative samples. Building such collections is also expensive unless they are already public, making these methods difficult to extend to newly released benchmarks. To address this challenge, we present ZipBench, a simple and low-cost BCM with theoretical error and rank-consistency guarantees. ZipBench evaluates only a small set of anchor LLMs, synthesizes pseudo evaluation results to broaden coverage, learns compact sample representations, and selects a small yet representative subset. Building on it, we create ZipBench Zoo, a collection of compact versions of 100+ benchmark proxies spanning text, multimodal, and agent tasks. These benchmark achieve mean absolute errors of 0.002--0.02 and average Spearman correlations of ~0.98 with the full benchmarks. Overall, ZipBench reduces the cost of both LLM evaluation and compact benchmark construction, lowering the barrier to broad LLM research for compute-constrained researchers. The code has been released in this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.12475 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T03:02:47.542Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).