Metraipolis › archive › dc_6f1671ceeebc

On the Feature Evolution of Temporal Difference Learning — arXiv:2611.04158

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.04158
License (as archived): CC0-1.0
Archived: 2026-11-04T12:40:45.576Z
SHA-256 of the archived text: 50d0cfb9a0f04ef76b86a00d7e5162e2081ebba81397705dacbf5e086763d0f6
Stable URL of this page: https://metraipolis.com/documento/dc_6f1671ceeebc

How to cite

On the Feature Evolution of Temporal Difference Learning — arXiv:2611.04158 — arXiv (en), https://arxiv.org/abs/2611.04158 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6f1671ceeebc (Metraipolis archive dc_6f1671ceeebc, sha256 50d0cfb9a0f04ef76b86a00d7e5162e2081ebba81397705dacbf5e086763d0f6)

BibTeX

@misc{dc6f1671ceeebc,
  title = {On the Feature Evolution of Temporal Difference Learning — arXiv:2611.04158},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04158},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6f1671ceeebc; sha256 50d0cfb9a0f04ef76b86a00d7e5162e2081ebba81397705dacbf5e086763d0f6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04158. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Temporal-difference (TD) learning trains neural representations using bootstrapped targets generated by evolving value estimates, yet how predictive features emerge and propagate under incomplete neural updates remains poorly understood. We investigate this problem using finite-width two-layer ReLU critics on a constructed family of smooth stochastic horizon-$H$ MDPs with unknown quadratic structure and $d$-dimensional observations. We first show that an informative TD update acquires the quadratic Bellman signal and yields a representation admitting a bounded predictive readout. The simultaneous hidden-weight and head update achieves a strict finite-sample prediction-risk improvement over matched head-only SGD, with a $2(d+1)$-fold larger population quadratic response. We then establish that an imperfect learned continuation can already provide informative supervision to an earlier zero-reward stage. By tracking coupled neural updates with one SGD step per target copy, without requiring intermediate Bellman regressions to be solved, we obtain a uniform prediction-error transient $\exp[-\Omega(K/(Hd))]$, up to action-independent shifts and explicit sampling and finite-width errors. On the constructed signed-action MDP family whose total reward is bounded by 1, we show that an exploration-then-commit algorithm can achieve $\widetilde{O}(d^3H^{1/3}T^{2/3})$ regret over $T$ episodes, using $K=\widetilde{O}(Hd)$ global updates. Our analysis connects bootstrapped feature acquisition, concurrent neural refinement, and finite-budget policy performance.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.04158 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:29:47.281Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).