Metraipolis › archive › dc_217c1937d405

TTPO: Test-Time Policy Optimization — arXiv:2608.27448

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.27448
License (as archived): CC0-1.0
Archived: 2026-10-17T15:20:45.883Z
SHA-256 of the archived text: 0971b08c590edb80743f35c121e860391c2c115f3f2d61ca5d47f8a2a893b6b5
Stable URL of this page: https://metraipolis.com/documento/dc_217c1937d405

How to cite

TTPO: Test-Time Policy Optimization — arXiv:2608.27448 — arXiv (en), https://arxiv.org/abs/2608.27448 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_217c1937d405 (Metraipolis archive dc_217c1937d405, sha256 0971b08c590edb80743f35c121e860391c2c115f3f2d61ca5d47f8a2a893b6b5)

BibTeX

@misc{dc217c1937d405,
  title = {TTPO: Test-Time Policy Optimization — arXiv:2608.27448},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27448},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_217c1937d405; sha256 0971b08c590edb80743f35c121e860391c2c115f3f2d61ca5d47f8a2a893b6b5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27448. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.27448 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T15:15:18.102Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).