Metraipolis › archive › dc_646b8060e08d

HARTS: Efficient Agentic Reinforcement Learning for Hybrid-Attention Models over Arbitrary Rollout Trees — arXiv:2608.28158

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.28158
License (as archived)
CC0-1.0
Archived
2026-10-17T10:56:51.160Z
SHA-256 of the archived text
56ac8892af6bb50c73e857265e8058ac121be0e3d8b3e6e8c654dc4cf9661980
Stable URL of this page
https://metraipolis.com/documento/dc_646b8060e08d

How to cite

HARTS: Efficient Agentic Reinforcement Learning for Hybrid-Attention Models over Arbitrary Rollout Trees — arXiv:2608.28158 — arXiv (en), https://arxiv.org/abs/2608.28158 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_646b8060e08d (Metraipolis archive dc_646b8060e08d, sha256 56ac8892af6bb50c73e857265e8058ac121be0e3d8b3e6e8c654dc4cf9661980)
BibTeX
@misc{dc646b8060e08d,
  title = {HARTS: Efficient Agentic Reinforcement Learning for Hybrid-Attention Models over Arbitrary Rollout Trees — arXiv:2608.28158},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.28158},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_646b8060e08d; sha256 56ac8892af6bb50c73e857265e8058ac121be0e3d8b3e6e8c654dc4cf9661980},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28158. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agentic reinforcement learning (RL) often produces irregular rollout trees with shared histories. Training root-to-leaf trajectories independently recomputes these shared prefixes. Existing systems primarily target full-attention models and lack dense, differentiable hybrid-attention execution compatible with activation recomputation. We present HARTS (Hybrid-Attention RL over Tree Structures). HARTS jointly plans microbatches, data-parallel (DP) replica assignments, and microbatch-slot schedules using non-replay compact-token work after prefix compression. For chunkwise linear attention, a linear-time algorithm coordinates chunk-boundary state recovery and replay and produces the minimum number of sequential linear-attention calls under our packed execution model. HARTS preserves the chunkwise state partitioning of trajectory-wise training: it does not repeat projections, MLP/MoE computation, or final outputs, and performs only bounded state replay for numerical alignment. Per round, HARTS batches all branches into one packed call, propagates gradients through differentiable state handoffs, supports activation recomputation, and restores per-token log-probabilities. For deterministic, no-token-drop top-$k$ MoE routing, semantic multiplicities restore MoE-objective token weights and load statistics. Existing RL objectives retain their interface. To our knowledge, HARTS is the first system to demonstrate arbitrary-rollout-tree prefix-sharing speedups on a real hybrid-attention model. On an Agentic RL workload generated from SWE-bench tasks, HARTS achieves $4.81$--$4.87\times$ forward/backward/gradient speedup with activation recomputation across multiple parallel configurations. Its numerical differences are comparable to baseline self-rerun variation, and its reward trend is similar to the baseline over the first 120 steps of $\tau^3$-Bench training.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-31 in cs.DC).
Categorie arXiv: Machine Learning (cs.LG) ; Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2608.28158 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:47:50.663Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).