Metraipolis › archive › dc_fb394b343dba

$T^5$: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training — arXiv:2609.32791

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32791
License (as archived)
CC0-1.0
Archived
2026-10-07T20:56:06.393Z
SHA-256 of the archived text
5549d82fe8be7207402c2f09141ca53d0a8c9963c98e8324cca3f6df900a22c8
Stable URL of this page
https://metraipolis.com/documento/dc_fb394b343dba

How to cite

$T^5$: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training — arXiv:2609.32791 — arXiv (en), https://arxiv.org/abs/2609.32791 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fb394b343dba (Metraipolis archive dc_fb394b343dba, sha256 5549d82fe8be7207402c2f09141ca53d0a8c9963c98e8324cca3f6df900a22c8)
BibTeX
@misc{dcfb394b343dba,
  title = {$T^5$: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training — arXiv:2609.32791},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32791},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fb394b343dba; sha256 5549d82fe8be7207402c2f09141ca53d0a8c9963c98e8324cca3f6df900a22c8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32791. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement mid-training lets language models learn internal thoughts from unlabeled text, but efficient token-level credit assignment remains challenging. Existing group-relative methods require costly repeated generation. Learned critics offer single-rollout feedback, but accurate return prediction alone does not ensure reliable policy updates. Our analysis shows how training--inference mismatch and PPO clipping prevent a common offset in advantage estimates from cancelling out, introducing additional update drift. We propose \tfour{}, a twin-critic method that calibrates token-level advantages from a single generated trajectory. After warmup and held-out qualification, the critics provide two advantage estimates, combined using action-dependent weights learned through a conditional-moment saddle-point objective. This objective brings the average advantage at each prefix toward zero, while a signal-retention constraint prevents the correction from erasing the learning signal. Sharing information across text positions avoids repeated sampling of each prefix. Theoretically, we characterize optimal mixing under the signal-retention constraint and establish an upper bound on residual mean-induced drift. Experiments show that, compared with the state-of-the-art critic-free method, \tfour{} improves mean benchmark performance by 7.8\% and reduces mean training-step time by up to 63.4\%.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.32791 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:00.979Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).