Metraipolis › archive › dc_4e625f98cf19

Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks — arXiv:2609.17817

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.17817
License (as archived): CC0-1.0
Archived: 2026-10-15T14:31:33.011Z
SHA-256 of the archived text: 528113e402ded07247188e46d5d829804c05bb0a9446a699621c7102c145179e
Stable URL of this page: https://metraipolis.com/documento/dc_4e625f98cf19

How to cite

Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks — arXiv:2609.17817 — arXiv (en), https://arxiv.org/abs/2609.17817 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4e625f98cf19 (Metraipolis archive dc_4e625f98cf19, sha256 528113e402ded07247188e46d5d829804c05bb0a9446a699621c7102c145179e)

BibTeX

@misc{dc4e625f98cf19,
  title = {Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks — arXiv:2609.17817},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.17817},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4e625f98cf19; sha256 528113e402ded07247188e46d5d829804c05bb0a9446a699621c7102c145179e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.17817. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Thompson's "Reflections on Trusting Trust" showed that a compiler can be poisoned to reinsert its own backdoor, so that even recompiling clean source reproduces the Trojan. Today, substantial coding work is done by AI coding agents -- and increasingly, those agents generate new versions of themselves. We reconsider Thompson's attack when the "compiler" is a self-modifying coding agent. Can an adversary supply poisoned benchmarks to the agent's self-evaluation and self-improvement process to induce future versions of the agent to write vulnerable code on clean, held-out tasks? We instantiate this attack against three recently proposed self-modifying coding agents: the Darwin Gödel Machine (with our experimental modifications), the Self-Improving Coding Agent, and Hyperagents (both substantively unmodified). We demonstrate successful proofs-of-concept: for example, with Hyperagents powered by Sonnet 4.5, our poisoned benchmark leads the agent to self-evolve instructions that disable HTTPS certificate validation on neutral URL-fetching tasks. From our experiments, we distill properties of the vulnerability, benchmark, model, and agent scaffolding that are sufficient to enable a benchmark poisoning attack. Moreover, we show that contamination often persists even when a poisoned agent is subsequently evolved against clean benchmarks. We discuss defensive directions and argue that self-modifying coding agents must be designed to be more resilient to such attacks.

Tema degli esploratori: fiducia digitale e crittografia (retro-catalogo arXiv: annunci del 2026-09-17 in cs.CR).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.17817 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:56:52.833Z dalla pagina https://arxiv.org/catchup/cs.CR/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).