Metraipolis › archive › dc_609e25ca0026
SIGMA: Self-Improving Alignment Generalization from a Model Spec — arXiv:2610.07935
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.07935
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T11:19:17.556Z
- SHA-256 of the archived text
1bd9c97c99f6db0f635fd4fc5c0fd614de59bd7edba4971fc47ee76b93b88ccf- Stable URL of this page
https://metraipolis.com/documento/dc_609e25ca0026
How to cite
SIGMA: Self-Improving Alignment Generalization from a Model Spec — arXiv:2610.07935 — arXiv (en), https://arxiv.org/abs/2610.07935 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_609e25ca0026 (Metraipolis archive dc_609e25ca0026, sha256 1bd9c97c99f6db0f635fd4fc5c0fd614de59bd7edba4971fc47ee76b93b88ccf)
BibTeX
@misc{dc609e25ca0026,
title = {SIGMA: Self-Improving Alignment Generalization from a Model Spec — arXiv:2610.07935},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.07935},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_609e25ca0026; sha256 1bd9c97c99f6db0f635fd4fc5c0fd614de59bd7edba4971fc47ee76b93b88ccf},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07935. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM agents are increasingly capable of executing complex tasks and of recursively improving themselves on easy-to-verify objectives such as software engineering and mathematics. Since alignment is much harder to verify, this creates a growing risk of capabilities increasing without appropriate safety alignment, especially as capabilities expand to auto-research and cybersecurity. Existing approaches focus on capability self-improvement using verifiable feedback or on alignment training with supervision from stronger models or curated data, creating an external supervision bottleneck for alignment. We ask whether current models can improve their own safety alignment, and propose SIGMA, a data generation and training pipeline enabling alignment self-improvement that generalizes to out-of-distribution settings. Given only a "Model Spec" stating the model's desired behavior, SIGMA leverages a model's reasoning capabilities to strengthen its own safety reasoning. SIGMA first performs spec-guided task synthesis, using the candidate model as a task designer agent to generate diverse alignment dilemma scenarios and convert them into training tasks that stress-test its understanding of the Model Spec. Next, SIGMA conducts self-judged alignment training through supervised fine-tuning and rubric-based reinforcement learning with the model itself as the reward model. Despite training only on single-turn chat data, SIGMA improves safety alignment in multi-turn agentic environments (AgentHarm harmfulness decreases from 22.6 to 14.8; Agentic Misalignment decreases from 79.1 to 3.8), outperforms Deliberative Alignment and Constitutional AI baselines, and retains general capability. Analyses show that a Model Spec balancing harmlessness and helpfulness, test-time reasoning for safety deliberation, and high-quality rubrics from SIGMA's task designer agent are crucial for effective self-improvement. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.07935 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T11:15:24.569Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).