Metraipolis › archive › dc_edaf4a696e95

Dory1.0-4B: A Looped Normalized-GPT — arXiv:2611.03998

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.03998
License (as archived)
CC0-1.0
Archived
2026-11-04T18:38:35.606Z
SHA-256 of the archived text
b70446e54e102288a54c6488e70fd220bc518479671ee2a84c9ed9a29d211cbc
Stable URL of this page
https://metraipolis.com/documento/dc_edaf4a696e95

How to cite

Dory1.0-4B: A Looped Normalized-GPT — arXiv:2611.03998 — arXiv (en), https://arxiv.org/abs/2611.03998 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_edaf4a696e95 (Metraipolis archive dc_edaf4a696e95, sha256 b70446e54e102288a54c6488e70fd220bc518479671ee2a84c9ed9a29d211cbc)
BibTeX
@misc{dcedaf4a696e95,
  title = {Dory1.0-4B: A Looped Normalized-GPT — arXiv:2611.03998},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03998},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_edaf4a696e95; sha256 b70446e54e102288a54c6488e70fd220bc518479671ee2a84c9ed9a29d211cbc},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03998. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We introduce Dory1.0-4B, a 4-billion-parameter looped (latent-recurrent) language model with elastic depth: a single checkpoint serves different inference compute budgets. Dory1.0-4B is built on a normalized Transformer (nGPT) backbone with interleaved sliding-window and full-attention layers, organized into an input block, a shared recurrent block, and an output block. At inference, the recurrent block can be applied a variable number of times. We pre-trained Dory1.0-4B on 6 trillion tokens. The first 4.5 trillion tokens were trained in NVFP4, and additional pre-training then extended the context length to 256K tokens. Next, we post-trained the model using Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR), with a difficulty-aware loop curriculum that routes harder prompts to deeper loops. Increasing the loop count improves accuracy, with the largest gains on coding, agentic, and long-context benchmarks. With two or more loops, Dory1.0-4B outperforms larger models such as Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B on average across 12 benchmarks. Looping also makes Dory1.0-4B more token-efficient on agentic benchmarks: with four loops, it generates 1.3-2.5x fewer tokens than with one loop, as the extra computation is spent in latent space rather than on longer outputs. To reduce memory during looping, Dory1.0-4B caches keys and values only from the last loop, keeping its KV-cache footprint equal to that of a non-looped model at any loop count while retaining most of the accuracy gains. We open-source the BF16, MXFP8, NVFP4, and GGUF quantized checkpoints.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.03998 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:32:52.130Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).