Metraipolis › archive › dc_fb705bb373bf

Memorization Diagnostics for Code LLMs Should be Scale-Aware — arXiv:2608.12771

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.12771
License (as archived): CC0-1.0
Archived: 2026-10-19T04:18:07.781Z
SHA-256 of the archived text: 724e25cdffb35fa855937861e55cea1e75bca6bfee34dbb24dbbda58d69a1efb
Stable URL of this page: https://metraipolis.com/documento/dc_fb705bb373bf

How to cite

Memorization Diagnostics for Code LLMs Should be Scale-Aware — arXiv:2608.12771 — arXiv (en), https://arxiv.org/abs/2608.12771 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fb705bb373bf (Metraipolis archive dc_fb705bb373bf, sha256 724e25cdffb35fa855937861e55cea1e75bca6bfee34dbb24dbbda58d69a1efb)

BibTeX

@misc{dcfb705bb373bf,
  title = {Memorization Diagnostics for Code LLMs Should be Scale-Aware — arXiv:2608.12771},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.12771},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fb705bb373bf; sha256 724e25cdffb35fa855937861e55cea1e75bca6bfee34dbb24dbbda58d69a1efb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12771. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The extent to which large language models for code rely on memorization over genuine understanding remains highly debated. While current literature frequently reports widespread memorization, evaluating the underlying probing techniques across dense architectures reveals a severe breakdown in their utility at scale. Traditional encoder-style probes using perturbations such as synonym fuzzing or dead-code insertion struggle to expose memorization in scaled models, even on known-contaminated benchmarks, and decoder-style probes that rely on log probabilities show similar performance degradation. The specific mode of failure for these probes, particularly why such techniques disrupt smaller models but fail to impact larger ones, motivates us to untangle representation load from memorization rather than treating them as a single phenomenon. By applying invertible mathematical transforms to numeric problems, we isolate these two factors and reveal that scaled encoders successfully absorb substantial representation load while still converging on the correct family of solutions. In practical software engineering, this ability to adapt to varying surface forms is what truly matters for usability and generalizability in LLM and agentic applications. Whether a specific solution was seen during training becomes a much less pressing question because although memorization inflates scores on contaminated benchmarks, factoring out representation load makes it debatable how much we should truly care if a functional answer was originally memorized. Future evaluations must therefore be built around separating these phenomena rather than relying on methodologies that quietly entangle them.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-14 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.12771 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T04:13:06.701Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).