Metraipolis › archive › dc_99c31f5ce5bc
Right Numbers, Wrong Questions: Evaluating LLMs on Underspecified Financial Queries — arXiv:2610.38631
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.38631
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T06:09:01.244Z
- SHA-256 of the archived text
1e762fcda7f0a6006e1c3dc5d9ff5892e709b34291cd9e199b400152e52e2b4e- Stable URL of this page
https://metraipolis.com/documento/dc_99c31f5ce5bc
How to cite
Right Numbers, Wrong Questions: Evaluating LLMs on Underspecified Financial Queries — arXiv:2610.38631 — arXiv (en), https://arxiv.org/abs/2610.38631 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_99c31f5ce5bc (Metraipolis archive dc_99c31f5ce5bc, sha256 1e762fcda7f0a6006e1c3dc5d9ff5892e709b34291cd9e199b400152e52e2b4e)
BibTeX
@misc{dc99c31f5ce5bc,
title = {Right Numbers, Wrong Questions: Evaluating LLMs on Underspecified Financial Queries — arXiv:2610.38631},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.38631},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_99c31f5ce5bc; sha256 1e762fcda7f0a6006e1c3dc5d9ff5892e709b34291cd9e199b400152e52e2b4e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.38631. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Financial assistants answer questions that sound factual but depend on what the user leaves unsaid: what a data figure measures, which product variant is meant, and the investor's situation and the rules that apply. An assistant that ignores these can still answer with a real number from its data, one that answers a different question. We call this frame collapse. We study it in Indian mutual funds, where product variants and tax rules are formally defined, so the correct answer can be computed for every reading. One study tests each challenge. Given an ambiguously labelled figure, models used it, often under the user's label. Given every alternative labelled, they listed them all, but the layout decided which came first. Missing a fact about the investor, weaker agents assumed a default or an outdated tax rule; requiring the fact did not guarantee its use. Because such answers match their data, checks against the data pass them. An AI judge caught mislabelled figures only when told what each field means, and still passed answers that silently chose another reading. We show how these failures arise, not how often, and propose controls for financial assistants. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); General Finance (q-fin.GN). Pagina del lavoro: https://arxiv.org/abs/2610.38631 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T06:07:08.486Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).