Metraipolis › archive › dc_f65698cd2a3f

Octopus: One Brain, Many Branches for Latent-KV Multi-Agent Reinforcement Learning — arXiv:2611.06956

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.06956
License (as archived): CC0-1.0
Archived: 2026-11-06T14:43:18.708Z
SHA-256 of the archived text: 09616417d820f3905184054eb5db5356d2c38f5a599ef8a54787c6049ae2e239
Stable URL of this page: https://metraipolis.com/documento/dc_f65698cd2a3f

How to cite

Octopus: One Brain, Many Branches for Latent-KV Multi-Agent Reinforcement Learning — arXiv:2611.06956 — arXiv (en), https://arxiv.org/abs/2611.06956 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f65698cd2a3f (Metraipolis archive dc_f65698cd2a3f, sha256 09616417d820f3905184054eb5db5356d2c38f5a599ef8a54787c6049ae2e239)

BibTeX

@misc{dcf65698cd2a3f,
  title = {Octopus: One Brain, Many Branches for Latent-KV Multi-Agent Reinforcement Learning — arXiv:2611.06956},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.06956},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f65698cd2a3f; sha256 09616417d820f3905184054eb5db5356d2c38f5a599ef8a54787c6049ae2e239},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06956. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multi-agent LLM systems explore in parallel by spawning sub-agents with isolated context windows, whereas single-agent systems append every search and verification step into one growing trajectory. However, existing multi-agent frameworks duplicate prefix KV caches across branches, exchange findings by re-prefilling text summaries, and flatten or drop cross-branch dependencies during reinforcement learning (RL). We introduce Octopus, a single-model multi-agent RL framework in which all threads share one backbone and coordinate over a GPU-resident directed acyclic graph (DAG) of KV states through five primitives (fork, read_thread, wait_threads, rewind, commit). Octopus combines three co-designed mechanisms: (1) Zero-Copy Radix Forking, which spawns branches as $\mathcal{O}(1)$ cursor views over shared physical KV pages; (2) Attention-Only KV Mounting, which mounts a sibling thread's terminal KV blocks directly into full-attention layers while bypassing linear-recurrent states (Gated DeltaNet / Mamba-2), cutting off-task hijacking by $21\times$ ($57.9\% \to 2.7\%$) while retaining $92.2\%$ of full-text mutual information at zero extra KV allocation; and (3) a Packed DAG-GRPO Trainer, which packs all episode branches into one block-sparse attention pass and assigns counterfactual branch credit via leave-one-branch-out marginal rewards. Trained end-to-end with GRPO on Qwen3.5-4B over a sparse BM25 corpus, Octopus reaches $49.6\%$ accuracy on BrowseComp-Plus ($+17.5$ points over single-agent GRPO at 143K context and $+12.5$ points over text-summary multi-agent RL) and $51.2\%$ on OpenResearcher, outperforming dense-retrieval agents up to $250\times$ larger while increasing training throughput $3.3\times$ and cutting peak KV memory $2.8\times$.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.06956 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:18:57.595Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).