Metraipolis › archive › dc_8d16a5fcc035

When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics — arXiv:2608.13835

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.13835
License (as archived)
CC0-1.0
Archived
2026-10-19T02:27:53.166Z
SHA-256 of the archived text
b84cfc6cfa4172563ac85f6f4470eb4468c258df0edcc772955d7061667158b1
Stable URL of this page
https://metraipolis.com/documento/dc_8d16a5fcc035

How to cite

When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics — arXiv:2608.13835 — arXiv (en), https://arxiv.org/abs/2608.13835 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8d16a5fcc035 (Metraipolis archive dc_8d16a5fcc035, sha256 b84cfc6cfa4172563ac85f6f4470eb4468c258df0edcc772955d7061667158b1)
BibTeX
@misc{dc8d16a5fcc035,
  title = {When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics — arXiv:2608.13835},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.13835},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8d16a5fcc035; sha256 b84cfc6cfa4172563ac85f6f4470eb4468c258df0edcc772955d7061667158b1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13835. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Dynamic topic models capture evolving word distributions, but traditional coherence metrics may fail when vocabulary changes while semantic meaning persists. We evaluate 120 topics from CoNTM and DLDA across NYT, DBLP, and arXiv, using three human annotators and Low, Medium, and High lexical-change categories. Traditional temporal coherence shows highly variable agreement with human judgments ($\rho$=-0.256 to 0.614). In contrast, LLM-based semantic similarity agrees strongly with human semantic judgments for CoNTM on NYT ($\rho$=0.609), DBLP ($\rho$=0.721), and arXiv ($\rho$=0.502), but is less consistent for DLDA. Lexical-change stratification reveals variation hidden by aggregate evaluation. We therefore advocate lexical-change-aware evaluation, jointly reporting traditional coherence and LLM-based semantic measures as complementary rather than interchangeable signals.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.13835 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:20:31.899Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).