Metraipolis › archive › dc_fdca1f92455e

LoopQ: Fair, Closed-Loop Evaluation of Quantized Vision-Language-Action Models — arXiv:2611.05093

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.05093
License (as archived)
CC0-1.0
Archived
2026-11-04T19:05:18.935Z
SHA-256 of the archived text
3d367829ac932778500b415d42e9710ca14728cc9449dbc275269363bab4c6b9
Stable URL of this page
https://metraipolis.com/documento/dc_fdca1f92455e

How to cite

LoopQ: Fair, Closed-Loop Evaluation of Quantized Vision-Language-Action Models — arXiv:2611.05093 — arXiv (en), https://arxiv.org/abs/2611.05093 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fdca1f92455e (Metraipolis archive dc_fdca1f92455e, sha256 3d367829ac932778500b415d42e9710ca14728cc9449dbc275269363bab4c6b9)
BibTeX
@misc{dcfdca1f92455e,
  title = {LoopQ: Fair, Closed-Loop Evaluation of Quantized Vision-Language-Action Models — arXiv:2611.05093},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05093},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fdca1f92455e; sha256 3d367829ac932778500b415d42e9710ca14728cc9449dbc275269363bab4c6b9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05093. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Post-training quantization (PTQ) for vision-language-action (VLA) models promises smaller and faster VLA deployments. However, when analyzing existing VLA quantization works, we realised that it is hard to reason about the performance of the suggested quantization method compared to existing, similar, or in many cases, simpler PTQ methods. We present LoopQ, a closed-loop benchmark that compares multiple quantization policies on the same checkpoint, layers and engine, varying only the method and the kernels. We evaluate RTN, AWQ, GPTQ, ActQuant and QuantVLA on four VLAs with more than 180,000 episodes of LIBERO and LIBERO-plus, compare against QVLA's reported results on the same layers, and benchmark whether the four public VLA-quantization releases can be reproduced. We benchmark the methods on an NVIDIA H100, an L4, and a Jetson AGX Thor. Our results show that while quantization reliably saves memory, 2-3x for real low-bit models, VLA-specific PTQ methods underperform compared to simpler generic baselines. At 4 bits, RTN, AWQ and GPTQ match or exceed ActQuant and QuantVLA, plain RTN matches the calibrated methods, and plain AWQ matches the success QVLA reports for itself. Standard LIBERO hides the losses that do exist: configurations with no significant LIBERO loss lose 1.5-4.3 points under paired LIBERO-plus perturbations. Speed is a weaker story. At batch 1 on the H100, no quantized pi0.5 or OpenVLA-OFT configuration is faster than its 16-bit model (1.06-1.58x slower on library kernels); only the L4 with int8 kernels gains (4-14%), the Thor is slower on every 4-bit configuration, and the engine alone changes latency by 2.1x. Profiling explains why: pi0.5's quantized backbone is compute-bound, while its memory-bound action expert, 63% of GPU time and 13% of the parameters, stays in 16 bits.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.RO, nuovi lavori del giorno).
Categorie arXiv: Robotics (cs.RO).
Pagina del lavoro: https://arxiv.org/abs/2611.05093 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:39:52.423Z dalla pagina https://arxiv.org/list/cs.RO/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).