Metraipolis › archive › dc_af4088bb09f0

Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model — arXiv:2609.23959

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.23959
License (as archived)
CC0-1.0
Archived
2026-10-09T06:50:00.389Z
SHA-256 of the archived text
e1aedeb3796d060c59cccc64358ef75f4b0e81a40f637826a0d5fd61252dc766
Stable URL of this page
https://metraipolis.com/documento/dc_af4088bb09f0

How to cite

Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model — arXiv:2609.23959 — arXiv (en), https://arxiv.org/abs/2609.23959 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_af4088bb09f0 (Metraipolis archive dc_af4088bb09f0, sha256 e1aedeb3796d060c59cccc64358ef75f4b0e81a40f637826a0d5fd61252dc766)
BibTeX
@misc{dcaf4088bb09f0,
  title = {Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model — arXiv:2609.23959},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.23959},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_af4088bb09f0; sha256 e1aedeb3796d060c59cccc64358ef75f4b0e81a40f637826a0d5fd61252dc766},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.23959. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Screening a phone call for fraud needs a trustworthy probability after every caller turn, in milliseconds. Jev-style typed decisions promise exactly that: declared options go in, one calibrated probability per option comes out of a single forward pass, with no generated text. We test an open implementation of this readout, JevLite, on scam-call screening: Qwen3-4B is LoRA-tuned so that the temperature-scaled softmax over two answer-label logits is P(scam). On 41 held-out CallScreenBench scenarios (577 per-turn decisions) a three-seed ensemble reaches AUROC .974 with calibration error .052, non-inferior to an LLM judge (MiniMax-M3) at a pre-registered .02 margin, with no false alarms on legitimate calls, decisions 1.14 turns earlier under the same hang-up rule, and 64.5 ms per decision on one consumer GPU, 4.9x lower than the same backbone fine-tuned to generate its answer. The gain is in the readout and calibration, not accuracy: a fine-tuned ModernBERT encoder is not significantly worse, the recipe was selected with test-set exposure, and all callers are synthetic. We claim no architectural novelty; the contribution is the application and an evaluation reporting calibration, false alarms and decision timing alongside AUROC.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.23959 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:48:45.020Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).