Metraipolis › archive › dc_0d024e0a809a

What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend — arXiv:2608.04714

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.04714
License (as archived): CC0-1.0
Archived: 2026-10-31T11:14:34.522Z
SHA-256 of the archived text: f9fb17e945a2e67908d26142a22716a84a4221b2729e72d6a8917d78bfe86387
Stable URL of this page: https://metraipolis.com/documento/dc_0d024e0a809a

How to cite

What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend — arXiv:2608.04714 — arXiv (en), https://arxiv.org/abs/2608.04714 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0d024e0a809a (Metraipolis archive dc_0d024e0a809a, sha256 f9fb17e945a2e67908d26142a22716a84a4221b2729e72d6a8917d78bfe86387)

BibTeX

@misc{dc0d024e0a809a,
  title = {What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend — arXiv:2608.04714},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.04714},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0d024e0a809a; sha256 f9fb17e945a2e67908d26142a22716a84a4221b2729e72d6a8917d78bfe86387},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04714. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this work we investigate how much this choice can influence the model output. In a fully-crossed study (three instruction-tuned models x five inference frameworks x six benchmarks x four generation modes) we investigate how different tools (wrappers/backend) influence benchmark scores and how their score changes is influenced by generation hyper-parameters. We find backend to be a non-negligible factor where even under greedy, sampling-noise-free decoding, changing the backend can significantly alter models performance and this effect is structural and strongly model-dependent. Decomposing the variance according to generation mode reveal that considerable portion of the variability (roughly 39\%) a practitioner sees out-of-the-box can stem from the backend, while the remaining stems from sampling noise and each framework's default generation parameters, both of which are avoidable by disclosing and matching the generation configuration. These divergences are more pronounced on factual than on social-bias benchmarks. Overall, benchmark numbers are not backend-agnostic therefore, we recommend disclosing the backend, its version, and the full generation configuration, also using deterministic decoding for cross-backend comparison.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.04714 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:02:13.544Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).