Metraipolis › archive › dc_1f734eb879bb

Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox — arXiv:2610.32955

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.32955
License (as archived)
CC0-1.0
Archived
2026-10-26T19:53:44.397Z
SHA-256 of the archived text
a95e8f80c6c93f928724dceb6bee361af5b4f0d9b190126f246f13936933e759
Stable URL of this page
https://metraipolis.com/documento/dc_1f734eb879bb

How to cite

Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox — arXiv:2610.32955 — arXiv (en), https://arxiv.org/abs/2610.32955 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1f734eb879bb (Metraipolis archive dc_1f734eb879bb, sha256 a95e8f80c6c93f928724dceb6bee361af5b4f0d9b190126f246f13936933e759)
BibTeX
@misc{dc1f734eb879bb,
  title = {Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox — arXiv:2610.32955},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.32955},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1f734eb879bb; sha256 a95e8f80c6c93f928724dceb6bee361af5b4f0d9b190126f246f13936933e759},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.32955. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Personal email agents decide, for every message, what it is, whether it is urgent, whether to draft a reply, create a task, or ignore it. Their error rate is rarely measured: the stored labels were written by the system itself. We present Foyer, a self-hosted personal inbox agent in daily use, and a measurement protocol (stratified sampling, frozen snapshots, thread-grouped splits, pre-registered hypotheses, human labels in the agent's review loop) that yields ground truth before any training. On 739 real Italian/English emails labelled by the owner (231 unbiased, 500 adversarially sampled), we compare nine systems under thread-grouped cross-validation: the production keyword rules, a TF-IDF + logistic-regression baseline, local typed decision models (Tev1, Nimble), a hosted decision model (Nimble v4-max), and hosted generative LLMs (Qwen2.5-7B, Gemma 4 26B/31B). Four results. The production rules score 28% on message type against a 58% majority class: the field shown most prominently carried no information. With ~200 labels, logistic regression has the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90), though it never predicts the two rarest classes and is statistically tied with Gemma 4 on type; few-shot prompting does not close the gap. The sender's relationship (client, supplier, service) is recovered from other emails of the same contact with 98.6% accuracy where the contact has been seen, and is not inferable from a single message by any model: it belongs in the contact graph, not the prompt. Finally, accuracy is not the product metric: composed into actions, the most accurate system most often ignores emails needing action (18% vs 4% for the best LLM); a cost-chosen threshold on its noise probability halves that rate and makes it the cheapest system.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Human-Computer Interaction (cs.HC); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.32955 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:45:48.376Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).