Metraipolis › archive › dc_ef8db5de12e7

Where Do Distilled Students Store Thinking? — arXiv:2611.04141

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.04141
License (as archived): CC0-1.0
Archived: 2026-11-04T21:42:22.661Z
SHA-256 of the archived text: 4e4afe8a40b72cfdcab40a60fa46047b3420639133898adcfdff469362ad7661
Stable URL of this page: https://metraipolis.com/documento/dc_ef8db5de12e7

How to cite

Where Do Distilled Students Store Thinking? — arXiv:2611.04141 — arXiv (en), https://arxiv.org/abs/2611.04141 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ef8db5de12e7 (Metraipolis archive dc_ef8db5de12e7, sha256 4e4afe8a40b72cfdcab40a60fa46047b3420639133898adcfdff469362ad7661)

BibTeX

@misc{dcef8db5de12e7,
  title = {Where Do Distilled Students Store Thinking? — arXiv:2611.04141},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04141},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ef8db5de12e7; sha256 4e4afe8a40b72cfdcab40a60fa46047b3420639133898adcfdff469362ad7661},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04141. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Small students degrade on large reasoning models' (LRMs') long solutions, so recent work shortens them by interpolating an LRM with a short-answer model at ratio $\lambda$. We ask what student weights retain when this teacher stops thinking, comparing LoRA adapters of three students from three teacher pairs at each $\lambda$. In each pair, the adapters' update subspace shifts most where the teacher stops thinking and solutions shorten most, locating the change without attributing it. We identify a shared adapter subspace tied to thinking-like behavior (reflective markers, monologue format and long solutions). Shared by adapters before that point, it keeps about half (post hoc) while one pair's teacher halves its solutions but keeps its think tag, and largely empties after thinking stops. Removing it largely turns this behavior off for every pair, student and benchmark. Only in that pair and one other student does it leave clearly fewer markers than removing the update's top components; elsewhere, same-energy random directions often nearly match it. Grafted into a short-answer student at twice its weight, it turns early markers (a post hoc count) on, like the whole update, but the student rarely finishes. Our work identifies a weight subspace tied to distilled thinking-like behavior.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.04141 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:36:04.409Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).