Metraipolis › archive › dc_00a114c2a680

The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations — arXiv:2608.18300

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.18300
License (as archived): CC0-1.0
Archived: 2026-10-18T13:44:26.184Z
SHA-256 of the archived text: 972560b673bc61aeb46145bb914c4af4d1fcad781c46460b51387e2004f8c409
Stable URL of this page: https://metraipolis.com/documento/dc_00a114c2a680

How to cite

The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations — arXiv:2608.18300 — arXiv (en), https://arxiv.org/abs/2608.18300 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_00a114c2a680 (Metraipolis archive dc_00a114c2a680, sha256 972560b673bc61aeb46145bb914c4af4d1fcad781c46460b51387e2004f8c409)

BibTeX

@misc{dc00a114c2a680,
  title = {The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations — arXiv:2608.18300},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.18300},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_00a114c2a680; sha256 972560b673bc61aeb46145bb914c4af4d1fcad781c46460b51387e2004f8c409},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18300. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM-as-a-Judge, which leverages a large language model to evaluate natural language generated by another AI application or model, has become a standard, scalable approach for accelerating and extending costly human evaluation. Yet most work treats a judge as a static artifact, evaluating it once at construction or against a fixed benchmark. We argue instead that an LLM judge operating in a deployed system is better understood as having a lifecycle. It must be built, trained, deployed, and continuously maintained as the surrounding data evolves, and each phase poses distinct technical and operational challenges. We present such a lifecycle for the LLM judges that evaluate recommendation explanations at Netflix. Everything we report comes out of a series of controlled online member-facing experiments, in which our pipeline generated and the judges assessed hundreds of thousands of distinct show-level explanations per week across a changing catalog. Our framework has four phases. (I) Birth defines the evaluation criteria and builds curated benchmark datasets with human labels and rationales. (II) Training refines the judges' rubrics via Reasoning-Aligned Rubric Tuning (RART), which uses a meta-judge over reasoning output as the learning signal. (III) Deployment puts one judge in two online roles, quality gating and reflective generation. (IV) Monitoring runs a continuous Human-in-the-Loop (HITL) alignment process that detects drift and triggers re-tuning behind a human review gate. We report results from a five-week online A/B test over tens of millions of members on the Netflix mobile app, in which judge-aligned explanations shifted member viewing toward novel content (previously unwatched) and increased successful browse-to-play sessions relative to a no-explanation control, with no quality-related escalations.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.18300 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:41:17.855Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).