Metraipolis › archive › dc_4a33335083b0

MemStrata: 95% and 90.91% Source-Aware Accuracy on LongMemEval-500 and LoCoMo-1540 with a Local Qwen 3.8 27B Q4_K_M Reader — arXiv:2610.05343

Source
arXiv
Original
https://arxiv.org/abs/2610.05343
License (as archived)
CC0-1.0
Archived
2026-10-06T05:28:19.663Z
SHA-256 of the archived text
5f20c5f4d7100dd2f19842f8d1b3144272b8567367b649690efa8f5ed4cfa335
Stable URL of this page
https://metraipolis.com/documento/dc_4a33335083b0

How to cite

MemStrata: 95% and 90.91% Source-Aware Accuracy on LongMemEval-500 and LoCoMo-1540 with a Local Qwen 3.8 27B Q4_K_M Reader — arXiv:2610.05343 — arXiv, https://arxiv.org/abs/2610.05343 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4a33335083b0 (Metraipolis archive dc_4a33335083b0, sha256 5f20c5f4d7100dd2f19842f8d1b3144272b8567367b649690efa8f5ed4cfa335)
BibTeX
@misc{dc4a33335083b0,
  title = {MemStrata: 95% and 90.91% Source-Aware Accuracy on LongMemEval-500 and LoCoMo-1540 with a Local Qwen 3.8 27B Q4_K_M Reader — arXiv:2610.05343},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05343},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4a33335083b0; sha256 5f20c5f4d7100dd2f19842f8d1b3144272b8567367b649690efa8f5ed4cfa335},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05343. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

An adequate conversational answer may differ from a short or incomplete benchmark reference. To measure adequacy against the recorded history we prefer source-aware grading, in which the judge checks the reference against the full source before assessing system-blinded answers; original reference-only grading is reported alongside. With a local Qwen 3.8 27B Q4_K_M reader and a 24,000-token evidence ceiling, MemStrata CL1 scores 475/500 (95.0%) on LongMemEval-S and 1,400/1,540 (90.91%) on LoCoMo categories 1-4 under source-aware GPT-5.5 adjudication, against 463/500 (92.6%) and 1,205/1,540 (78.25%) under reference-only grading of the same answers. It preserves a retrieval backbone and adds nonduplicated, dated, speaker-attributed source spans. A same-reader full-history control with about 4.7 times the evidence scores 464/500 reference-only and 470/500 (94.0%) source-aware; neither difference is decisive. Keyword-only selection at the same budget scores 425, and a matched-reader Letta arm 438. On LongMemEval-M, where the packet holds about 1.6% of each history, MemStrata CL1 scores 427/500, with losses concentrated in multi-session and temporal questions. On 300 BEAM-1M questions it outscores dense retrieval, 0.738 to 0.706 (Wilcoxon p = 0.011). A same-seed replay of unchanged requests changed 1.5-2.3% of labels. On identical packets GLM 5.3 flash is non-inferior within 3 points (462 versus 463); Muse Spark 1.3 did not show non-inferiority on 269 questions. None of four pre-registered interventions met all of its registered advancement or feasibility criteria. Signed read-side artifacts support inspection but do not regenerate the private retrieval pipeline. The superiority of source-aware grading to human adjudication is not established, and development exposure, automated-judge dependence and the absence of held-out data preclude an independent-replication or leaderboard claim.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2610.05343 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:20.326Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).