Metraipolis › archive › dc_7f07f5edbfe0

High-Level Text Preprocessing for Semantic Similarity Analysis of Discursive Texts: A Framework and Empirical Demonstration — arXiv:2609.33983

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.33983
License (as archived)
CC0-1.0
Archived
2026-10-07T21:53:16.017Z
SHA-256 of the archived text
d8c832a64826b0cf233cd84f4d73decc78fc7694263d693b2f723c652593f7a8
Stable URL of this page
https://metraipolis.com/documento/dc_7f07f5edbfe0

How to cite

High-Level Text Preprocessing for Semantic Similarity Analysis of Discursive Texts: A Framework and Empirical Demonstration — arXiv:2609.33983 — arXiv (en), https://arxiv.org/abs/2609.33983 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7f07f5edbfe0 (Metraipolis archive dc_7f07f5edbfe0, sha256 d8c832a64826b0cf233cd84f4d73decc78fc7694263d693b2f723c652593f7a8)
BibTeX
@misc{dc7f07f5edbfe0,
  title = {High-Level Text Preprocessing for Semantic Similarity Analysis of Discursive Texts: A Framework and Empirical Demonstration — arXiv:2609.33983},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33983},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7f07f5edbfe0; sha256 d8c832a64826b0cf233cd84f4d73decc78fc7694263d693b2f723c652593f7a8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33983. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Semantic Textual Similarity (STS) methods assume that a document's lexical content faithfully represents what it asserts. This assumption fails for discursive documents that discuss, compare, critique, and contextualize other positions in the process of articulating their own. The result is semantic diffusion: similarity scores between documents are inflated by vocabulary acquired through discursive engagement rather than substantive alignment. Standard Natural Language Processing (NLP) preprocessing (tokenization, stopword removal, stemming, lemmatization) cannot address this problem because it operates at the lexical level, treating all content identically regardless of its discursive function. This paper introduces high-level text preprocessing: a systematic, rule-based intervention applied before the standard preprocessing pipeline to isolate each document's actual claim from its discursive structure. We propose 12 rules, each with an explicit rationale, and demonstrate their effect on an encyclopedic philosophical corpus: three entries from the Stanford Encyclopedia of Philosophy (virtue ethics, deontological ethics, and consequentialism). A three-phase experiment using eight Transformer-based STS models shows that preprocessing reduces centroid cosine similarity scores across all three theory pairs, with 23 of 24 model-pair comparisons showing the expected decrease and cross-model agreement ranging from 7-1 to 8-0. We introduce the semantic diffusion index (SDI), a per-document metric for assessing the semantic reorientation between a document's raw and high-level preprocessed representations. Although the framework is demonstrated using philosophical texts, it potentially addresses a domain-agnostic problem applicable to legal texts, policy documents, academic articles, and any genre in which a discursive approach introduces vocabulary from positions the document does not endorse.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Information Retrieval (cs.IR); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.33983 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T21:27:08.575Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).