Metraipolis › archive › dc_032fc12cc2d8

World Modeling in Transformers — arXiv:2609.21748

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.21748
License (as archived): CC0-1.0
Archived: 2026-10-10T01:05:28.013Z
SHA-256 of the archived text: 20cee870a69ba1677bd8358fa9a2f7b27b6505abdd15a33df537cead80b0209a
Stable URL of this page: https://metraipolis.com/documento/dc_032fc12cc2d8

How to cite

World Modeling in Transformers — arXiv:2609.21748 — arXiv (en), https://arxiv.org/abs/2609.21748 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_032fc12cc2d8 (Metraipolis archive dc_032fc12cc2d8, sha256 20cee870a69ba1677bd8358fa9a2f7b27b6505abdd15a33df537cead80b0209a)

BibTeX

@misc{dc032fc12cc2d8,
  title = {World Modeling in Transformers — arXiv:2609.21748},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.21748},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_032fc12cc2d8; sha256 20cee870a69ba1677bd8358fa9a2f7b27b6505abdd15a33df537cead80b0209a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21748. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Behavioral failures can make a transformer appear to lack a world model even when it has learned faithful representations of its environment. We demonstrate this in TaxiGPT, a transformer trained on random walks through Manhattan whose failures have been interpreted as evidence of an incoherent internal map. Through mechanistic analysis and causal interventions, we show that the model represents intersections and streets, tracks its position, and uses a goal compass to navigate. We trace its failures to interference between superposed intersection features, which disrupts localization within the internal map. Affordance packing, which groups representations of intersections with the same legal moves, helps limit the consequences of these errors. Finally, we propose mechanistic indicators that we use to compare models and show that world-modeling capacities emerge at different stages of training. Our findings motivate a shift from asking whether a model has a world model to mechanistically studying its world modeling: the interacting capacities through which it represents its environment and uses those representations to guide behavior.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.21748 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T00:23:42.473Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).