Metraipolis › archive › dc_722a773ece60

The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention — arXiv:2610.00910

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00910
License (as archived)
CC0-1.0
Archived
2026-10-06T12:11:52.853Z
SHA-256 of the archived text
02058cc2d20dc0b36939877bceaa92e083a7d24ecd80912de9891c63a88792ac
Stable URL of this page
https://metraipolis.com/documento/dc_722a773ece60

How to cite

The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention — arXiv:2610.00910 — arXiv (en), https://arxiv.org/abs/2610.00910 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_722a773ece60 (Metraipolis archive dc_722a773ece60, sha256 02058cc2d20dc0b36939877bceaa92e083a7d24ecd80912de9891c63a88792ac)
BibTeX
@misc{dc722a773ece60,
  title = {The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention — arXiv:2610.00910},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00910},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_722a773ece60; sha256 02058cc2d20dc0b36939877bceaa92e083a7d24ecd80912de9891c63a88792ac},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00910. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Human reasoning depends on how objects are related within propositions. \textit{How do relations organize the language representations of contextual contents?} We give an LLM a list of facts in its context (e.g., \emph{Alice eats an apple. Bob eats a pear.}) and measure how its hidden state changes when the question switches from what Alice eats to what Bob eats. Averaged over many lists, this change is a steering vector, which we call the \emph{ordinal vector}. It points to a fact by its \emph{order of mention}, the order in which the facts were stated in the context. We find that LLMs represent the fact a question asks about by its order of mention, not by the name the question contains. We state this as the \textit{ordinal addressing hypothesis}: each order of mention has a \emph{fact address} in the model's state, shared by all contexts, and a question moves the state to the fact address of the fact it asks about, while the context supplies what that fact says. Across Qwen, Gemma, and Llama, fact addresses are (1) \emph{ordered by mention}: query states are organized by the order of facts, not of names, even when one fact has multiple subjects; (2) \emph{steerable}: added to a question about the first fact of a new list, the ordinal vector makes the model answer with the second fact of that list; (3) \emph{low-rank}: they span a low-rank subspace in which the first-mentioned fact is the easiest to reach, surprisingly similar to human recall; and (4) \emph{emergent}: they are shared in late-middle layers, hold from 1.5B to 32B parameters, and form early in pretraining. Language models reach a stated fact by where it was mentioned, deepening our understanding of LLM reasoning.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.00910 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.131Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).