Metraipolis › archive › dc_3cfb2563cbd9
Circuit Oracle: Automating Attribution Graph Analysis via Natural-Language Queries — arXiv:2611.00420
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00420
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:09:42.656Z
- SHA-256 of the archived text
9659243593f488c87ab07762cd0f06eca9ee642ace84a74555c87e1b3f090a76- Stable URL of this page
https://metraipolis.com/documento/dc_3cfb2563cbd9
How to cite
Circuit Oracle: Automating Attribution Graph Analysis via Natural-Language Queries — arXiv:2611.00420 — arXiv (en), https://arxiv.org/abs/2611.00420 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3cfb2563cbd9 (Metraipolis archive dc_3cfb2563cbd9, sha256 9659243593f488c87ab07762cd0f06eca9ee642ace84a74555c87e1b3f090a76)
BibTeX
@misc{dc3cfb2563cbd9,
title = {Circuit Oracle: Automating Attribution Graph Analysis via Natural-Language Queries — arXiv:2611.00420},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00420},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3cfb2563cbd9; sha256 9659243593f488c87ab07762cd0f06eca9ee642ace84a74555c87e1b3f090a76},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00420. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Attribution graphs, an emerging tool in mechanistic interpretability, use transcoders to decompose language model computations into sparse interpretable features connected by causal edges. However, turning a graph into a safety-relevant insight requires hours of manual analysis by experts. We introduce Circuit Oracle, an off-the-shelf LLM equipped with a multi-agent harness and task-specific authored skills, which answers natural-language questions about a target model by reading the attribution graph through tool calls. We evaluate it on three safety-relevant proxy tasks: detecting spurious features in probe circuits, eliciting hidden knowledge from taboo-finetuned models, and suppression jailbreaking via feature-level interventions. Beyond task scores, the oracle produces pruned subgraphs and context-specific feature explanations that we assess qualitatively. Results are mixed. On spurious-feature detection, Circuit Oracle reaches 78.2 $\pm$ 2.4% accuracy, outperforming single-layer feature-ranking baselines. On secret elicitation, trained self-explainers like Activation Oracles remain the stronger baseline. On refusal suppression over 50 prompts, its single committed intervention is slightly below a diff-in-means baseline, and only exceeds it when allowed the best of five attempts. Results are stable across five runs and across model choices for the agents. We also release the pruned subgraphs and context-specific feature readings that revise existing global labels, and invite future work to develop evaluations that can measure their usefulness directly. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2611.00420 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:59:52.253Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).