Metraipolis › archive › dc_e63ac79990c8
Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization — arXiv:2608.30315
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.30315
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T08:08:09.534Z
- SHA-256 of the archived text
8e4948adcd25076dfe9ec09c169097ab450b3ab462337b1f77cfd23fd35f6f33- Stable URL of this page
https://metraipolis.com/documento/dc_e63ac79990c8
How to cite
Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization — arXiv:2608.30315 — arXiv (en), https://arxiv.org/abs/2608.30315 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e63ac79990c8 (Metraipolis archive dc_e63ac79990c8, sha256 8e4948adcd25076dfe9ec09c169097ab450b3ab462337b1f77cfd23fd35f6f33)
BibTeX
@misc{dce63ac79990c8,
title = {Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization — arXiv:2608.30315},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.30315},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e63ac79990c8; sha256 8e4948adcd25076dfe9ec09c169097ab450b3ab462337b1f77cfd23fd35f6f33},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.30315. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Token embeddings are the basic representational units that connect discrete tokens with continuous computation in language models. Although modern language models learn embeddings from random initialization through gradient-based training, the dynamical mechanism by which meaningful embedding structures emerge remains unclear. In this work, we identify that the evolving embedding structures are closely related to token-conditioned label and contextual distributions, which we formalize as probability signatures. We observe a progressive learning process, which we term Context Staircase: embeddings learn the low-order statistic signatures of the data before the high-order ones. More specifically, we observe that early in training they align with the simplest, context-free signature linking a token to its label, and as training proceeds, they progressively reflect signatures involving more and more context tokens. We then analyze the gradient flow of embeddings under small initialization to explain this phenomenon, deriving embedding evolution equations for feed-forward and self-attention architectures. We further extend these observations to real language-model training. Finally, we show that these embedding structures play an important role in both task learning and the incorporation of semantic structure into the embedding space. Overall, our results provide a dynamic explanation of how data statistics and architecture jointly shape token embeddings in language models, and reveal an implicit bias in the space of data statistics: training proceeds from simpler, low-order statistical relations toward increasingly complex, context-dependent ones. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.30315 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:47:52.922Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).