Metraipolis › archive › dc_84429ed503d2

A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation — arXiv:2609.01315

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.01315
License (as archived)
CC0-1.0
Archived
2026-10-17T06:32:25.511Z
SHA-256 of the archived text
7625cb87a0fdf3f385e6d70a4e76e3f66061387332e427a2193c5818e87a4a25
Stable URL of this page
https://metraipolis.com/documento/dc_84429ed503d2

How to cite

A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation — arXiv:2609.01315 — arXiv (en), https://arxiv.org/abs/2609.01315 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_84429ed503d2 (Metraipolis archive dc_84429ed503d2, sha256 7625cb87a0fdf3f385e6d70a4e76e3f66061387332e427a2193c5818e87a4a25)
BibTeX
@misc{dc84429ed503d2,
  title = {A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation — arXiv:2609.01315},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.01315},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_84429ed503d2; sha256 7625cb87a0fdf3f385e6d70a4e76e3f66061387332e427a2193c5818e87a4a25},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01315. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Building an omni-modal foundation model means evaluating it across text, image, video, and audio. Excellent evaluation toolkits exist for each modality, but their inference engines, prompt conventions, and metric implementations are mutually incompatible, so practitioners end up maintaining separate environments for every toolchain and still struggle to compare results across them. OmniEvaluator grew out of this need in our own model development: rather than reimplementing benchmarks, it connects existing inference engines and curated evaluation libraries at a higher level, exposing four inference backends, four evaluation frameworks, and over a thousand benchmarks through a single interface. Every run is recorded as an artifact capturing the full configuration for exact reproduction, and results flow into a shared dashboard for cross-model comparison. A federated mode shares GPU inference servers across concurrent evaluations, and a built-in verifier, small enough to run on CPU, keeps its score stable across engines and prompts where rule-based scoring fluctuates under configuration mismatch, matching cost-efficient commercial LLM judges without their recurring API cost. The system, demo video, and dashboard are publicly available. ( this https URL )

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.01315 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:15:12.687Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).