Metraipolis › archive › dc_4ae831769e69

On the Feature Evolution of Temporal Difference Learning — arXiv:2611.04158

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.04158
License (as archived): CC0-1.0
Archived: 2026-11-04T21:51:29.005Z
SHA-256 of the archived text: 7fb7f4d0cdd3f3253a0e2c083572bc6c65ab48f3779ee23652c2724764368dec
Stable URL of this page: https://metraipolis.com/documento/dc_4ae831769e69

How to cite

On the Feature Evolution of Temporal Difference Learning — arXiv:2611.04158 — arXiv (en), https://arxiv.org/abs/2611.04158 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4ae831769e69 (Metraipolis archive dc_4ae831769e69, sha256 7fb7f4d0cdd3f3253a0e2c083572bc6c65ab48f3779ee23652c2724764368dec)

BibTeX

@misc{dc4ae831769e69,
  title = {On the Feature Evolution of Temporal Difference Learning — arXiv:2611.04158},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04158},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4ae831769e69; sha256 7fb7f4d0cdd3f3253a0e2c083572bc6c65ab48f3779ee23652c2724764368dec},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04158. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Temporal-difference (TD) learning trains neural representations using bootstrapped targets generated by evolving value estimates, yet how predictive features emerge and propagate under incomplete neural updates remains poorly understood. We investigate this problem using finite-width two-layer ReLU critics on a constructed family of smooth stochastic horizon-$H$ MDPs with unknown quadratic structure and $d$-dimensional observations. We first show that an informative TD update acquires the quadratic Bellman signal and yields a representation admitting a bounded predictive readout. The simultaneous hidden-weight and head update achieves a strict finite-sample prediction-risk improvement over matched head-only SGD, with a $2(d+1)$-fold larger population quadratic response. We then establish that an imperfect learned continuation can already provide informative supervision to an earlier zero-reward stage. By tracking coupled neural updates with one SGD step per target copy, without requiring intermediate Bellman regressions to be solved, we obtain a uniform prediction-error transient $\exp[-\Omega(K/(Hd))]$, up to action-independent shifts and explicit sampling and finite-width errors. On the constructed signed-action MDP family whose total reward is bounded by 1, we show that an exploration-then-commit algorithm can achieve $\widetilde{O}(d^3H^{1/3}T^{2/3})$ regret over $T$ episodes, using $K=\widetilde{O}(Hd)$ global updates. Our analysis connects bootstrapped feature acquisition, concurrent neural refinement, and finite-budget policy performance.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.04158 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:38:49.704Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).