Metraipolis › archive › dc_cd3d3f4e302e

Enhancing Language Model Reasoning with Structured Multi-Level Modeling — arXiv:2611.00616

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.00616
License (as archived): CC0-1.0
Archived: 2026-11-03T18:22:37.660Z
SHA-256 of the archived text: 01f74f7b297e18df80cfc020a0d89deeb05b6be46ce101bc111468e6f464f442
Stable URL of this page: https://metraipolis.com/documento/dc_cd3d3f4e302e

How to cite

Enhancing Language Model Reasoning with Structured Multi-Level Modeling — arXiv:2611.00616 — arXiv (en), https://arxiv.org/abs/2611.00616 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cd3d3f4e302e (Metraipolis archive dc_cd3d3f4e302e, sha256 01f74f7b297e18df80cfc020a0d89deeb05b6be46ce101bc111468e6f464f442)

BibTeX

@misc{dccd3d3f4e302e,
  title = {Enhancing Language Model Reasoning with Structured Multi-Level Modeling — arXiv:2611.00616},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00616},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cd3d3f4e302e; sha256 01f74f7b297e18df80cfc020a0d89deeb05b6be46ce101bc111468e6f464f442},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00616. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Inference-time scaling enhances a model's reasoning by extending its chain-of-thought (CoT). However, existing approaches typically rely on a single policy trained with outcome-reward reinforcement learning (RL), which often suffers from long-horizon plan failures where the implicit plan drifts from valid strategies, especially for small LMs with limited capacity. To address this, we propose Multi-Level Reasoning (MLR), which reformulates long-CoT generation as a two-level stochastic process. A high-level planner generates structured step descriptors specifying both the reasoning mode and the semantic subgoal. The low-level executor then produces detailed reasoning conditioned on these descriptors, forming an alternating plan--execute loop. To maintain scalability, we adopt a minimal design where the base model serves as the low-level policy and a lightweight LoRA module implements the high-level policy. For training, we observe that outcome-reward RL provides sparse and delayed feedback for long trajectories (e.g., several thousand tokens), hindering credit assignment. We therefore introduce iterative Step-DPO, a process-level preference optimization scheme that leverages Twisted Sequential Monte Carlo (TSMC) to provide scalable stepwise supervision. This yields more effective training, improved stability, and higher accuracy. Extensive experiments on challenging math, science, and logical reasoning benchmarks show that, under the same reduced data budget (10% SFT and 5% preference relative to the DeepSeek-R1 distillation setup), MLR outperforms both SFT-based distillation and strong RL/preference-optimization baselines across multiple base models and tasks. Moreover, MLR exhibits slower performance degradation on long-horizon reasoning, demonstrating stronger robustness under extended CoT generation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.00616 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:10:07.893Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).