Metraipolis › archive › dc_04a3a2f11902

Inadvertent Context Leakage in Language Models — arXiv:2608.19857

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.19857
License (as archived)
CC0-1.0
Archived
2026-10-18T11:48:17.901Z
SHA-256 of the archived text
fac123d63daed2a950dd3e5f1f8b0cc294f24bab4abaa0cdc97fd51b8d66192a
Stable URL of this page
https://metraipolis.com/documento/dc_04a3a2f11902

How to cite

Inadvertent Context Leakage in Language Models — arXiv:2608.19857 — arXiv (en), https://arxiv.org/abs/2608.19857 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_04a3a2f11902 (Metraipolis archive dc_04a3a2f11902, sha256 fac123d63daed2a950dd3e5f1f8b0cc294f24bab4abaa0cdc97fd51b8d66192a)
BibTeX
@misc{dc04a3a2f11902,
  title = {Inadvertent Context Leakage in Language Models — arXiv:2608.19857},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.19857},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_04a3a2f11902; sha256 fac123d63daed2a950dd3e5f1f8b0cc294f24bab4abaa0cdc97fd51b8d66192a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.19857. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data. We study whether the mere presence of such secrets in a model's context window introduces hidden correlations into the model's benign outputs, allowing reconstruction even when the model correctly refuses direct extraction. We further study whether an adversary can actively engineer prompts that amplify this effect, using the model as a covert carrier to transmit secrets through seemingly innocuous text. In both cases, this limited leakage is exploited using a novel adaptive attack that assumes black-box access to the underlying model. In controlled experiments across eight proprietary models, we find that 2-digit in-context secrets are reconstructed with near-perfect accuracy and 4-digit secrets at 82\% exact match, all from outputs the model produces in response to ordinary, non-adversarial requests. We observe that more capable models leak more: stronger instruction-following amplifies sensitivity to in-context secrets, suggesting leakage is a byproduct of capability as opposed to a patchable bug. We show this leakage enables two practical attacks: (1) a trained classifier that infers semantic predicates about user memories (e.g., health conditions, financial events) from routine natural-language outputs, and (2) an RL-trained adversary that extracts full Social Security Numbers from a production-style agent.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-21 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Cryptography and Security (cs.CR).
Pagina del lavoro: https://arxiv.org/abs/2608.19857 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T11:47:43.321Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).