Metraipolis › archive › dc_6fa52ca4dc1c

A Definition of Catastrophic Performance — arXiv:2611.06219

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.06219
License (as archived)
CC0-1.0
Archived
2026-11-05T18:16:57.929Z
SHA-256 of the archived text
f53fff6d06bb0ccfd3fa8cd5aa11c06fd8bf5eef77c1d8d280a330c1bb35bd35
Stable URL of this page
https://metraipolis.com/documento/dc_6fa52ca4dc1c

How to cite

A Definition of Catastrophic Performance — arXiv:2611.06219 — arXiv (en), https://arxiv.org/abs/2611.06219 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6fa52ca4dc1c (Metraipolis archive dc_6fa52ca4dc1c, sha256 f53fff6d06bb0ccfd3fa8cd5aa11c06fd8bf5eef77c1d8d280a330c1bb35bd35)
BibTeX
@misc{dc6fa52ca4dc1c,
  title = {A Definition of Catastrophic Performance — arXiv:2611.06219},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.06219},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6fa52ca4dc1c; sha256 f53fff6d06bb0ccfd3fa8cd5aa11c06fd8bf5eef77c1d8d280a330c1bb35bd35},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06219. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A central challenge in AI is specifying objectives that align with human preferences. While existing theoretical results elucidate how misspecification impacts performance, they typically do not distinguish between merely suboptimal and catastrophic outcomes. To enable rigorous analysis of catastrophic risk from AI misalignment, we introduce a formal definition of catastrophic performance. Our definition is based on two reference values. The first, baseline value, captures the value attained if no agent is deployed. The second, orthogonal value, captures the value of a trajectory that is unrelated to human preferences, in a sense we describe. Orthogonal value is typically far worse than baseline value. We interpret these reference values through examples in which past human activity has altered the state of the environment to human benefit. Starting at the resulting state, an AI agent is deployed with the goal of accelerating further progress. Catastrophic performance represents this agent's undoing of benefits afforded by human activity.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.06219 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:12:49.007Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).