Metraipolis › archive › dc_2f60a1b0d044
On Scope Classification and Current Knowledge-Editing Benchmarks: A Negative Result, with INLAY as a Gradient-Free Case Study — arXiv:2608.26292
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.26292
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T14:24:40.733Z
- SHA-256 of the archived text
c3c68fc132e574e8133ae47c5566386e5378ebd826828a5c85eaa006bde2304d- Stable URL of this page
https://metraipolis.com/documento/dc_2f60a1b0d044
How to cite
On Scope Classification and Current Knowledge-Editing Benchmarks: A Negative Result, with INLAY as a Gradient-Free Case Study — arXiv:2608.26292 — arXiv (en), https://arxiv.org/abs/2608.26292 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2f60a1b0d044 (Metraipolis archive dc_2f60a1b0d044, sha256 c3c68fc132e574e8133ae47c5566386e5378ebd826828a5c85eaa006bde2304d)
BibTeX
@misc{dc2f60a1b0d044,
title = {On Scope Classification and Current Knowledge-Editing Benchmarks: A Negative Result, with INLAY as a Gradient-Free Case Study — arXiv:2608.26292},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.26292},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2f60a1b0d044; sha256 c3c68fc132e574e8133ae47c5566386e5378ebd826828a5c85eaa006bde2304d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26292. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Every memory-based knowledge editor in the SERAC lineage depends on a scope decision: given a query, does a stored edit apply? We report that current knowledge-editing benchmarks cannot measure this decision at all. Using INLAY, a gradient-free editor we built to obtain exact per-query ground truth (the model is frozen, edits live in an external addressable memory, and applying an edit is a bias added along one token's unembedding direction at decode time), we execute every candidate router action on 1,689 queries spanning three datasets and three input conditions. An oracle router choosing the best action every time ties a one-line static policy to four decimal places in all nine dataset-by-condition cells: the maximum attainable gain of any per-query routing method is 0.00 points. Abstention is the sole winning action zero times out of 1,689. The cause is structural: these are counterfactual benchmarks whose evaluation question asks for the post-edit answer, so answering from parametric knowledge is wrong by construction, and a benchmark without negatives cannot reward a classifier's ability to reject. This generalizes beyond our system to the whole scope-classifier family the benchmarks are used to evaluate. We confirm the mechanism directly: constructing the missing condition ourselves, by withholding a query's own edit from the index for half the sample, moves pooled headroom from exactly +0.0000 to +0.0420 and gives abstention its first wins. We also report where INLAY itself does not win (WISE beats it on Qwen2.5-7B CounterFact, and retrieval-augmented generation beats every method we tested, INLAY included, on rigorously matched RippleEdits), and disclose two bugs found during a self-audit of our own routing machinery, neither of which changed a published headline number outside noise. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.26292 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T13:16:53.624Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).