Metraipolis › archive › dc_4597ce1f7269

Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation — arXiv:2609.01604

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.01604
License (as archived)
CC0-1.0
Archived
2026-10-17T06:41:42.401Z
SHA-256 of the archived text
466d328e41dd19f4b439a43f4714fdfbce8dc59de5e813197303917ff2cea0d2
Stable URL of this page
https://metraipolis.com/documento/dc_4597ce1f7269

How to cite

Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation — arXiv:2609.01604 — arXiv (en), https://arxiv.org/abs/2609.01604 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4597ce1f7269 (Metraipolis archive dc_4597ce1f7269, sha256 466d328e41dd19f4b439a43f4714fdfbce8dc59de5e813197303917ff2cea0d2)
BibTeX
@misc{dc4597ce1f7269,
  title = {Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation — arXiv:2609.01604},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.01604},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4597ce1f7269; sha256 466d328e41dd19f4b439a43f4714fdfbce8dc59de5e813197303917ff2cea0d2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01604. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the internal procedure by which they assign a rating remains poorly understood. We investigate this procedure mechanistically through an eight-attack perturbation taxonomy across the Readability and Adequacy dimensions of NLG quality, a generation pipeline that produces paired clean and corrupt summaries with controlled error intensity and explicit token-level modification maps, and a four-experiment battery of causal tracing, logit-lens vocabulary projection, and attention-head knockout applied to Themis (Llama-3-8B) and Prometheus (Mistral-7B). Both evaluators implement a structured, coherent evaluation pipeline operating in two stages: below layer 15, attention performs local error comparison and routes the result to the final input position; above it, the MLP cascade integrates the signal and writes the rating, with the decision crystallizing in the residual stream at a sharp late layer (L = 26 on Themis, L = 25 on Prometheus). Furthermore, a base-model control at the same scale (Llama-3-8B) reproduces the routing architecture and crystallization but not the stage separation, isolating the two mechanisms that fine-tuning specifically installs, suppression of below-L15 MLP contribution at the last position and a two-layer advance of the crystallization depth, indicating that fine-tuning sculpts an existing substrate rather than building the pipeline from scratch. We release the source code and data at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.01604 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:33:38.878Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).