Metraipolis › archive › dc_414abceed495

Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions — arXiv:2608.31108

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.31108
License (as archived): CC0-1.0
Archived: 2026-10-17T08:39:32.315Z
SHA-256 of the archived text: df09f01c257b12694a0f112a88bac44c92589f08d853b6b0a6588eda87de47f4
Stable URL of this page: https://metraipolis.com/documento/dc_414abceed495

How to cite

Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions — arXiv:2608.31108 — arXiv (en), https://arxiv.org/abs/2608.31108 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_414abceed495 (Metraipolis archive dc_414abceed495, sha256 df09f01c257b12694a0f112a88bac44c92589f08d853b6b0a6588eda87de47f4)

BibTeX

@misc{dc414abceed495,
  title = {Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions — arXiv:2608.31108},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.31108},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_414abceed495; sha256 df09f01c257b12694a0f112a88bac44c92589f08d853b6b0a6588eda87de47f4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.31108. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Efficient evaluation changes the protocol used to support claims about model behavior, yet it is rarely tested whether those claims remain stable after the evaluation itself is made cheaper. We stress-test conclusion robustness in responsible-AI benchmarking by evaluating three dense and mixture-of-experts models on BBQ and BBQ-V under seven conditions spanning batching, quantization, benchmark reduction, and their combinations. Rather than treating preserved aggregate accuracy as sufficient, we compare accuracy, bias severity and prevalence, reasoning quality, subgroup behavior, subset-membership stability, runtime, and measured GPU energy against a full-benchmark BF16 baseline. Larger batching keeps accuracy within 0.35 percentage points of baseline and produces comparatively small subgroup changes, while reducing energy in five of six model--dataset settings. INT8 largely preserves quality but uses 1.79--4.26$\times$ baseline energy. INT4 causes larger, model- and context-dependent changes. Reduced benchmarks provide the most consistent savings, but very small subsets are substantially more sensitive to which items are retained. Efficient evaluation should therefore be treated as a measurement intervention whose validity must be checked across the conclusions the benchmark is intended to support. Our project website is this https URL and the code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.31108 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T08:13:26.868Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).