Metraipolis › archive › dc_fb69c2944285

SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models — arXiv:2608.27857

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27857
License (as archived)
CC0-1.0
Archived
2026-10-17T10:47:15.667Z
SHA-256 of the archived text
634789af05ee143d61944ae9d2dbf3383a7c1b2e404162b1a7cd1c1acbff1545
Stable URL of this page
https://metraipolis.com/documento/dc_fb69c2944285

How to cite

SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models — arXiv:2608.27857 — arXiv (en), https://arxiv.org/abs/2608.27857 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fb69c2944285 (Metraipolis archive dc_fb69c2944285, sha256 634789af05ee143d61944ae9d2dbf3383a7c1b2e404162b1a7cd1c1acbff1545)
BibTeX
@misc{dcfb69c2944285,
  title = {SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models — arXiv:2608.27857},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27857},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fb69c2944285; sha256 634789af05ee143d61944ae9d2dbf3383a7c1b2e404162b1a7cd1c1acbff1545},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27857. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Spiking neural networks (SNNs) offer a path to energy-efficient language modeling through sparse encoding and event-driven computation, but training capable spiking language models from scratch remains difficult. A practical alternative is ANN-to-SNN migration through knowledge distillation (KD), where a pretrained artificial neural network (ANN) teacher supervises an SNN student. Existing migration approaches distill on fixed corpus prefixes, whereas autoregressive inference conditions on self-generated prefixes, creating prefix-source mismatch. It manifests as output-policy mismatch with the ANN teacher and internal spiking-dynamics drift between self-generated and matched corpus prefixes. On-policy distillation (OPD) offers a natural way to mitigate both manifestations by continuing teacher supervision on self-generated prefixes. We evaluate a teacher-only full-KL variant, Vanilla OPD, via a controlled stress test and observe it may suffer from delayed rollout-feedback collapse. This result shows that on-policy coverage alone does not ensure stable adaptation. Motivated by these findings, we propose SpikeOPD, a stable on-policy distillation framework for autoregressive SNNs that learns from self-generated prefixes while maintaining rollout stability. It applies full-KL teacher correction to reduce output-policy mismatch, while matched-prefix policy anchoring constrains policy departure from the frozen reference SNN on the same prefixes. Layerwise spike regularization further limits firing-rate deviations during on-policy adaptation. Across three model scales, SpikeOPD improves average accuracy over the corresponding KD SNNs by 0.8, 1.7, and 2.9 points at 0.125B, 0.35B, and 1.3B, respectively, while preserving their sparse-compute profiles.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.27857 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T10:33:42.357Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).