Metraipolis › archive › dc_bbdc6fb8799c

Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference — arXiv:2609.08189

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.08189
License (as archived): CC0-1.0
Archived: 2026-10-16T12:57:52.562Z
SHA-256 of the archived text: 95270a96ddc91906ab7590082583c646d5b8b8c7588fc3850c25c68d8cdd3981
Stable URL of this page: https://metraipolis.com/documento/dc_bbdc6fb8799c

How to cite

Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference — arXiv:2609.08189 — arXiv (en), https://arxiv.org/abs/2609.08189 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bbdc6fb8799c (Metraipolis archive dc_bbdc6fb8799c, sha256 95270a96ddc91906ab7590082583c646d5b8b8c7588fc3850c25c68d8cdd3981)

BibTeX

@misc{dcbbdc6fb8799c,
  title = {Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference — arXiv:2609.08189},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.08189},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bbdc6fb8799c; sha256 95270a96ddc91906ab7590082583c646d5b8b8c7588fc3850c25c68d8cdd3981},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.08189. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens. Existing methods, however, treat each routing decision as a local operation conditioned solely on the current hidden state which is a formulation that overlooks the sequential, path-dependent nature of routing across depth: earlier decisions shape the representations seen by downstream routers, and the layer-usage objective couples all decisions jointly. We propose History-Aware Routing (HeRo), a dynamic routing framework that resolves this mismatch by introducing a router memory mechanism to maintain an explicit routing state across model depth. The memory is constructed via linear attention, incrementally aggregating preceding routing scores and their induced residual updates into a compact history representation. At each routed layer, the router conditions jointly on this accumulated state and the current hidden representation to select the executed branch. Instantiated for token-wise FFN routing, HeRo trains only lightweight routers and adapters on a frozen backbone, requiring no modification to pretrained parameters. Across Llama 3.1-8B, Llama 2-7B, and Llama 2-13B, HeRo consistently achieves the highest aggregate performance retention among ten baselines. On Llama 3.1-8B, it bypasses 26.87% of model parameters while achieving 100.24% of dense model performance across seven benchmarks, and retains 97.01% while bypassing 38.82% of model parameters under a tighter computation budget. Ablation studies confirm that removing routing history consistently degrades performance, most notably on multistep reasoning and code generation, validating that explicit routing memory enables more accurate and adaptive dynamic routing than solely conditioning on hidden state.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.08189 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T12:30:25.343Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).