Metraipolis › archive › dc_2eb40523208a
Benchmarking locally hosted language models for journal editorial work on a compact desktop workstation — arXiv:2609.11972
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.11972
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T03:11:40.864Z
- SHA-256 of the archived text
5282f6024f6743ce391ce63489e43f66bbb211224d5c153003d19f716017144d- Stable URL of this page
https://metraipolis.com/documento/dc_2eb40523208a
How to cite
Benchmarking locally hosted language models for journal editorial work on a compact desktop workstation — arXiv:2609.11972 — arXiv (en), https://arxiv.org/abs/2609.11972 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2eb40523208a (Metraipolis archive dc_2eb40523208a, sha256 5282f6024f6743ce391ce63489e43f66bbb211224d5c153003d19f716017144d)
BibTeX
@misc{dc2eb40523208a,
title = {Benchmarking locally hosted language models for journal editorial work on a compact desktop workstation — arXiv:2609.11972},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.11972},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2eb40523208a; sha256 5282f6024f6743ce391ce63489e43f66bbb211224d5c153003d19f716017144d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.11972. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Journals are beginning to consider language models for manuscript handling, but submitted manuscripts are unpublished, and where policy forbids sending them to an external service the model must run on hardware the journal controls. The capability of locally hosted models on editorial work has not been measured. Here we constructed a benchmark of eight editorial tasks from a journal's Instructions for Authors, from manuscripts carrying defects we seeded and verified independently, and from published reviews of a preprint, and evaluated twenty open-weight models spanning a twenty-five-fold range of weight size on a compact desktop workstation of the kind a laboratory or small editorial office can adopt. The strongest model detected 36 of 40 seeded guideline violations and occupied 81 GB; a 17 GB model detected 33. Across the best configurations tested, we observed no consistent monotonic association between weight size and score: rank correlations were negligible on every task (Spearman |rho| <= 0.19), and within one model family the larger member scored below its smaller sibling. A deterministic checker of regular expressions and arithmetic, using no model, detected 31 of the same violations in a fraction of a second, and the union of its detections with those of the strongest model covered all 40. On the single peer-review case, the best model recovered 6 of 12 points from three published reviews. Prompt structure substantially altered scores within individual models. This level of performance is therefore within reach of a workstation of this class, once the deterministic checks are written. Tema degli esploratori: ricerca e scienza aperta (retro-catalogo arXiv: annunci del 2026-09-14 in cs.DL). Categorie arXiv: Digital Libraries (cs.DL). Pagina del lavoro: https://arxiv.org/abs/2609.11972 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T03:04:37.202Z dalla pagina https://arxiv.org/catchup/cs.DL/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).