Metraipolis › archive › dc_c91f37b138bf

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes — arXiv:2609.10016

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.10016
License (as archived)
CC0-1.0
Archived
2026-10-16T07:26:44.154Z
SHA-256 of the archived text
2a0e3d201d7b6613d72cd58e4561868920ea647465b43ee2b659f5eda95bcb36
Stable URL of this page
https://metraipolis.com/documento/dc_c91f37b138bf

How to cite

MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes — arXiv:2609.10016 — arXiv (en), https://arxiv.org/abs/2609.10016 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c91f37b138bf (Metraipolis archive dc_c91f37b138bf, sha256 2a0e3d201d7b6613d72cd58e4561868920ea647465b43ee2b659f5eda95bcb36)
BibTeX
@misc{dcc91f37b138bf,
  title = {MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes — arXiv:2609.10016},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.10016},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c91f37b138bf; sha256 2a0e3d201d7b6613d72cd58e4561868920ea647465b43ee2b659f5eda95bcb36},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.10016. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk. It covers six real metro systems, ranging from 37 to 414 stations, and eleven categories that include routing, fare calculation, disruptions, accessibility, and adversarial input. In each case, the model must call structured tools and submit a machine-renderable terminal state containing an outcome, a per-ticket fare quote when applicable, and a kiosk action. Fourteen deterministic scoring components form Tier 1; eight semantic-quality components form Tier 2, six of which use a language-model judge. We report Tier 1 and the combined score of both tiers. A stratified 75/25 split reserves 717 cases for training-data generation and 238 for held-out evaluation. We evaluate twenty-six models from six vendors, of which twenty-three are ranked. On the held-out partition, a 4B Qwen 3.5 student trained through parameter-efficient fine-tuning (PEFT) exceeds both GPT-5.6 tiers on Tier 1 (91.3 against 90.6 and 90.0) and matches GPT-5.4 full at maximum reasoning effort (91.4), with a 2.6 GB Q4_K_M footprint. Larger 9B and 27B students provide no further Tier 1 improvement over the 4B student at this training scale. Across the four Qwen sizes, the PEFT gain over the corresponding base model decreases from +7.03 points at 2B (three training seeds) to -0.91 at 27B; every seed shows the same direction at every size. A deterministic rule-based baseline reaches 84.6 on Tier 1, with the remaining language-model advantage concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning. Muse Glimmer 30B leads the composite ranking, and serving configuration alone moves the Qwen 3.5-to-3.8 comparison by 2.7 Tier 1 points. The benchmark, harness, reproduction guide, and fine-tuned students are released at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-10 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.10016 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T07:00:16.372Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).