Metraipolis › archive › dc_19b9ea122ce5

MOSS: A Production Framework for Monitoring and Diagnosing Semantic Failures in Search and Recommendation Systems — arXiv:2610.26902

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.26902
License (as archived)
CC0-1.0
Archived
2026-10-21T07:07:16.823Z
SHA-256 of the archived text
e8a03f6e1f5eaaaa6869dd8b84abe8119a93aeca9b12cbab3eab2ebb4a801dc9
Stable URL of this page
https://metraipolis.com/documento/dc_19b9ea122ce5

How to cite

MOSS: A Production Framework for Monitoring and Diagnosing Semantic Failures in Search and Recommendation Systems — arXiv:2610.26902 — arXiv (en), https://arxiv.org/abs/2610.26902 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_19b9ea122ce5 (Metraipolis archive dc_19b9ea122ce5, sha256 e8a03f6e1f5eaaaa6869dd8b84abe8119a93aeca9b12cbab3eab2ebb4a801dc9)
BibTeX
@misc{dc19b9ea122ce5,
  title = {MOSS: A Production Framework for Monitoring and Diagnosing Semantic Failures in Search and Recommendation Systems — arXiv:2610.26902},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.26902},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_19b9ea122ce5; sha256 e8a03f6e1f5eaaaa6869dd8b84abe8119a93aeca9b12cbab3eab2ebb4a801dc9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.26902. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Detecting semantic failures in production recommendation and search systems is hard: behavioral metrics lag by days, manual review does not scale, and most LLM evaluation frameworks are designed for offline, fixed datasets rather than continuous operational monitoring. We present Model Sanity Service (MOSS), a production framework that addresses this operational gap by embedding LLM-based relevance scoring into a recurring monitoring loop with targeted sampling, strategy-aware prompting, structured alerting, and diagnostic follow-up prompting that categorizes failures into actionable root causes. MOSS has been running in production at a large U.S. retailer for over six months across recommendation, search, and offer surfaces, and has surfaced two confirmed serving defects. For one of these, a backfill blending defect, MOSS quantified the impact of the proposed fix offline as a 76.4\% relative reduction in alert rate. On a 498-pair human-annotated recommendation benchmark, frontier LLM judges achieve higher F1 and AUC point estimates than non-LLM baselines (internal production embeddings and SBERT), with the largest gains on strategy-dependent tasks requiring intent-level relational reasoning. The same judge, score scale, and threshold transfer to a 466-pair search benchmark spanning 21 queries with only a surface-specific prompt adaptation, achieving strong F1 and AUC without retuning. These results show that a single LLM monitoring framework can provide useful, actionable quality signals across multiple product discovery surfaces in a large-scale e-commerce setting.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Information Retrieval (cs.IR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.26902 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-21T07:05:43.503Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).