Metraipolis › archive › dc_ea1346fa850a

ChartBmkAgent: Harness-Governed Multi-Agent Construction of Chart QA Benchmarks from Sparse Error-Taxonomy Specifications — arXiv:2610.08106

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08106
License (as archived)
CC0-1.0
Archived
2026-10-07T11:19:27.095Z
SHA-256 of the archived text
04ee90f369303f36ff88617237ce4db0bfe6613fb79e1e7b0985f4fe007ae11f
Stable URL of this page
https://metraipolis.com/documento/dc_ea1346fa850a

How to cite

ChartBmkAgent: Harness-Governed Multi-Agent Construction of Chart QA Benchmarks from Sparse Error-Taxonomy Specifications — arXiv:2610.08106 — arXiv (en), https://arxiv.org/abs/2610.08106 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ea1346fa850a (Metraipolis archive dc_ea1346fa850a, sha256 04ee90f369303f36ff88617237ce4db0bfe6613fb79e1e7b0985f4fe007ae11f)
BibTeX
@misc{dcea1346fa850a,
  title = {ChartBmkAgent: Harness-Governed Multi-Agent Construction of Chart QA Benchmarks from Sparse Error-Taxonomy Specifications — arXiv:2610.08106},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08106},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ea1346fa850a; sha256 04ee90f369303f36ff88617237ce4db0bfe6613fb79e1e7b0985f4fe007ae11f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08106. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multimodal large language models (MLLMs) advance rapidly, while conventional benchmark development lags behind, delaying investigation of newly observed capability gaps. Such investigation requires an expressive task format and an on-demand construction process: information-rich charts make chart question answering (Chart QA) suitable for probing coupled perception and reasoning. Automated Chart QA construction is intended to shorten the benchmark-development cycle by turning identified gaps into targeted samples on demand. Current methods, however, commonly separate target guidance from scratch generation: target-guided systems often require prepared data, charts, or templates, while scratch-generation systems primarily ensure artifact validity, without explicitly controlling whether newly synthesized requirements and content remain aligned with an externally specified diagnostic target. We introduce ChartBmkAgent, which turns an identified capability gap into targeted diagnostic evidence by constructing complete Chart QA samples from sparse error-taxonomy specifications. Throughout construction, a central harness governs specialized agents, requires stage-specific evidence of alignment with the original error category, and records the basis for each acceptance decision. On 300 taxonomy-wide samples, MLLM accuracies ranged from 32.7% to 84.3% with distinct category profiles, showing that generated samples reveal capability differences. Across three source-model comparisons, targeted follow-ups scored 50.0% versus 82.2% on matched controls ($p=8.96\times10^{-6}$); all six cross-model comparisons had the same direction, demonstrating targeted validation and diagnostic-data generation. Multiple evaluator models assessed whether each sample tested its specified error category; 86.4% met this criterion, providing empirical evidence of target preservation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.08106 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:24.588Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).