Metraipolis › archive › dc_b40ff93fc13a
Target-Token Asymmetry Can Masquerade As Depth Specialization — arXiv:2610.19344
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.19344
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T14:23:19.702Z
- SHA-256 of the archived text
4d186ba2314bbff1cb7b39a47e2732c1ce6a77bbcc57ab45528e573b640ec64e- Stable URL of this page
https://metraipolis.com/documento/dc_b40ff93fc13a
How to cite
Target-Token Asymmetry Can Masquerade As Depth Specialization — arXiv:2610.19344 — arXiv (en), https://arxiv.org/abs/2610.19344 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b40ff93fc13a (Metraipolis archive dc_b40ff93fc13a, sha256 4d186ba2314bbff1cb7b39a47e2732c1ce6a77bbcc57ab45528e573b640ec64e)
BibTeX
@misc{dcb40ff93fc13a,
title = {Target-Token Asymmetry Can Masquerade As Depth Specialization — arXiv:2610.19344},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.19344},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b40ff93fc13a; sha256 4d186ba2314bbff1cb7b39a47e2732c1ce6a77bbcc57ab45528e573b640ec64e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19344. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Layer-ablation studies often perturb early or late layers of a language model, compare the damage to prediction on two corpora, and read a larger early-layer effect on one corpus as early-layer specialization for its task. Such a comparison averages over the tokens that each corpus asks the model to predict, and these target tokens differ: math solutions, for example, ask for far more whitespace and numbers than encyclopedia prose. We show that this difference can create the appearance of depth specialization. We split a cross-corpus early--late damage interaction by target-token type and recompute it under a target mixture shared by both corpora. Between GSM8K and Wikitext, a positive interaction replicates across five model instances. In the three fully decomposed models, only whitespace has a positive interaction. Removing whitespace reverses the pooled effect in all three, and matching the full target mixture reverses it in two. The diagnosis holds under position matching, adversarial reassignment of token types, and a continuous readout. The risk can also be anticipated from the tokenizer alone. A screen computed before any model runs places every held-out pair it scored in the region its label predicts, and on the pairs it rates at high or medium risk, removing whitespace reverses every positive pooled estimate. A smaller content-word effect one token ahead follows the share of content targets that the unablated model already predicts. A cross-corpus depth effect therefore supports an interpretation of task specialization only after the target-token mixture is reported, the effect is decomposed by token type, and a mixture-matched summary keeps its sign. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.19344 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T13:53:35.848Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).