Metraipolis › archive › dc_c41c0915efa3

Can Open-Weight Models Compete on Financial Text Comprehension? — arXiv:2608.08634

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.08634
License (as archived)
CC0-1.0
Archived
2026-10-19T08:50:50.359Z
SHA-256 of the archived text
b723ecc4ee6ecdae280463534f8aee9ed5c1316fdce0f391a47f0636dcbed3ac
Stable URL of this page
https://metraipolis.com/documento/dc_c41c0915efa3

How to cite

Can Open-Weight Models Compete on Financial Text Comprehension? — arXiv:2608.08634 — arXiv (en), https://arxiv.org/abs/2608.08634 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c41c0915efa3 (Metraipolis archive dc_c41c0915efa3, sha256 b723ecc4ee6ecdae280463534f8aee9ed5c1316fdce0f391a47f0636dcbed3ac)
BibTeX
@misc{dcc41c0915efa3,
  title = {Can Open-Weight Models Compete on Financial Text Comprehension? — arXiv:2608.08634},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.08634},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c41c0915efa3; sha256 b723ecc4ee6ecdae280463534f8aee9ed5c1316fdce0f391a47f0636dcbed3ac},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.08634. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Open-weight language models from Chinese AI labs caught up on benchmarks relative to proprietary frontier models in recent months. Yet their reliability on real-world financial tasks remains largely untested. We updated the Financial Touchstone benchmark, which now has 2,967 question context-answer triplets across 495 international annual reports. We also apply a new set of models on the benchmark, expanding coverage from eleven to twenty models across ten providers, including recent open-weight models such as GLM 4.7, GLM 5, Kimi K2.6, and DeepSeek V3.2, as well as Alibaba's proprietary flagship Qwen3-Max. Anthropic's Claude Opus 4.6 achieves the highest accuracy (88.4%), while Google's Gemini 2.5 Pro maintains the lowest hallucination rate (0.08%). Notably, the open-weight Kimi K2.6 ranks third in accuracy, and the non-reasoning models GLM 5 and Mistral 3 rank fourth and fifth, challenging the assumption that reasoning architectures or proprietary weights are a prerequisite for strong financial comprehension. Information retrieval remains the primary bottleneck, accounting for 48.9% of all failures. We also document a new finding: geopolitical content filters in Chinese models refuse legitimate financial questions (0.08% of attempts), sometimes without clear reason, and the refusal behavior depends on the access route as much as on the model. The complete dataset and evaluation framework are publicly available.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-11 in cs.CL).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Information Retrieval (cs.IR); General Finance (q-fin.GN).
Pagina del lavoro: https://arxiv.org/abs/2608.08634 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:49:41.990Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).