Metraipolis › archive › dc_565b1eddef26
Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval — arXiv:2610.02438
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.02438
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-05T22:50:01.122Z
- SHA-256 of the archived text
718c69f1fc2a1a98b9df6cbc0be8f157fa46b61143b183612d839335057d2c2e- Stable URL of this page
https://metraipolis.com/documento/dc_565b1eddef26
How to cite
Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval — arXiv:2610.02438 — arXiv, https://arxiv.org/abs/2610.02438 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_565b1eddef26 (Metraipolis archive dc_565b1eddef26, sha256 718c69f1fc2a1a98b9df6cbc0be8f157fa46b61143b183612d839335057d2c2e)
BibTeX
@misc{dc565b1eddef26,
title = {Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval — arXiv:2610.02438},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.02438},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_565b1eddef26; sha256 718c69f1fc2a1a98b9df6cbc0be8f157fa46b61143b183612d839335057d2c2e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02438. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models (LLMs) have demonstrated strong performance in code generation, where success depends on both recalling relevant algorithmic knowledge and reasoning about how to apply it. However, existing LLM pipelines are opaque, with no explicit separation between these two components. We argue that for well-known algorithms whose canonical implementations are widely accessible in pretraining corpora, code generation is better measured as \textit{parametric code retrieval}: reproducing a named algorithm from internalised knowledge rather than synthesizing a novel one. We introduce AlgoREval, a benchmark of 599 problems spanning classical 77 algorithms across 14 domains, 7 programming languages, and 4 graph-input representations to evaluate this capability in isolation, and assess 15 models (7B--34B parameters) in a zero-shot setting. We find substantial variation in retrieval accuracy across languages and input representations, even for widely documented algorithms and show that prompt augmentation with retrieved code snippets or structured algorithmic hints improve accuracy on complex algorithms, while SFT achieves broader language gains and GRPO achieves larger per-language gains on specific languages. Together, our results establish parametric code retrieval as a distinct, measurable capability and caution against deploying AI-generated algorithmic code without systematic validation.\footnote{Code and dataset are available at this https URL
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Programming Languages (cs.PL).
Pagina del lavoro: https://arxiv.org/abs/2610.02438 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:45:55.755Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).