Metraipolis › archive › dc_f460288237a7

When Less is More: Understanding When Token Filtering Helps and Fails in AI-generated Text Detection — arXiv:2608.29903

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.29903
License (as archived)
CC0-1.0
Archived
2026-10-17T07:52:16.235Z
SHA-256 of the archived text
6957100396b2b6ce4c152fb50d73cf2db31b988a1b5b6e94f0ff8b665cd0f253
Stable URL of this page
https://metraipolis.com/documento/dc_f460288237a7

How to cite

When Less is More: Understanding When Token Filtering Helps and Fails in AI-generated Text Detection — arXiv:2608.29903 — arXiv (en), https://arxiv.org/abs/2608.29903 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f460288237a7 (Metraipolis archive dc_f460288237a7, sha256 6957100396b2b6ce4c152fb50d73cf2db31b988a1b5b6e94f0ff8b665cd0f253)
BibTeX
@misc{dcf460288237a7,
  title = {When Less is More: Understanding When Token Filtering Helps and Fails in AI-generated Text Detection — arXiv:2608.29903},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.29903},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f460288237a7; sha256 6957100396b2b6ce4c152fb50d73cf2db31b988a1b5b6e94f0ff8b665cd0f253},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29903. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The rapid advancement of large language models (LLMs) has made AI-generated text detection increasingly critical. Existing zero-shot detectors assume that more token-level evidence leads to more reliable detection. However, our empirical study challenges this consensus: fewer tokens sometimes work better, retaining only 40% can yield optimal performance, yet this benefit is not universal. Using the Entropy Gap Score (EGS), we introduce top-$k$ cumulative probability filtering as a diagnostic probe. Across three representative settings, filtering exhibits strikingly different behaviors. We analyze EGS via typical set theory and quantify its dynamics through entropy calibration and distribution analysis. We find that filtering helps for weak source LMs, where low-entropy tokens are harmful, but fails for strong source LMs, where they are not notably harmful. Our work provides the first systematic analysis showing that some tokens are not merely uninformative but systematically harmful due to entropy miscalibration, revealing a two-sided trade-off in token-level detection.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.29903 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.802Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).