Metraipolis › archive › dc_a44835a3deab

Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning — arXiv:2610.00526

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00526
License (as archived)
CC0-1.0
Archived
2026-10-06T12:09:52.908Z
SHA-256 of the archived text
c19fd8310f5a6e272e2e587e923026ec4c78597e98a58e21ef8a546adc16573c
Stable URL of this page
https://metraipolis.com/documento/dc_a44835a3deab

How to cite

Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning — arXiv:2610.00526 — arXiv (en), https://arxiv.org/abs/2610.00526 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a44835a3deab (Metraipolis archive dc_a44835a3deab, sha256 c19fd8310f5a6e272e2e587e923026ec4c78597e98a58e21ef8a546adc16573c)
BibTeX
@misc{dca44835a3deab,
  title = {Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning — arXiv:2610.00526},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00526},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a44835a3deab; sha256 c19fd8310f5a6e272e2e587e923026ec4c78597e98a58e21ef8a546adc16573c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00526. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In-context learning (ICL) can be amortized into latent objects (task vectors, function vectors, context vectors) that recover few-shot behavior at zero-shot inference cost, but recent theory shows a static vector acts as a single synthetic demonstration and must fail on high-rank mappings such as word-level bijections. We ask a linguistic version of this question: which linguistic operations can be amortized out of the prompt? We train a 2.6M-parameter network that reads the geometry of a few-shot support set (centroid, principal subspace, spectrum, computed once and cached) and produces an input-conditioned additive update to the query's residual stream at a mid-depth layer of a frozen GPT-2-large/XL. Across eight inflectional directions and one lexical relation, under a canonical split that bars inverted-pair leakage between directions, three regimes emerge. On forward inflection, where 10-shot ICL is strong (0.67-0.89) and extracted task vectors collapse (<=0.06), the transform matches ICL at strictly zero-shot per-query cost. On lemmatization directions, which frozen GPT-2 can execute but 10 demonstrations systematically fail to convey (ICL 0.13-0.48 at 1.5B), the transform is not capped by ICL at all: it reaches 0.78-0.92, up to +72 points over ICL (past to present: 0.85 vs. 0.13). On arbitrary pairings (antonymy) every amortizer plateaus near half of ICL at every scale, capacity, and seed tested. Controls show the support manifold acts as a causally necessary task fingerprint: wrong-task manifolds collapse accuracy to <=0.06, query-only variants cannot disambiguate tasks sharing an input space, and leave-one-task-out transfer is zero. Productive rules amortize into latent task representations, sometimes better than prompting can convey them; memorized pairings do not.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.00526 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.077Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).