Metraipolis › archive › dc_bca271cced59

onóna: Twin Embeddings with Tied Unembeddings — arXiv:2611.10556

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.10556
License (as archived): CC0-1.0
Archived: 2026-11-10T18:44:27.893Z
SHA-256 of the archived text: ff96172375eaf83420fd49cba8832d48a354ceb93ff709c9ed73718d09285ae5
Stable URL of this page: https://metraipolis.com/documento/dc_bca271cced59

How to cite

onóna: Twin Embeddings with Tied Unembeddings — arXiv:2611.10556 — arXiv (en), https://arxiv.org/abs/2611.10556 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bca271cced59 (Metraipolis archive dc_bca271cced59, sha256 ff96172375eaf83420fd49cba8832d48a354ceb93ff709c9ed73718d09285ae5)

BibTeX

@misc{dcbca271cced59,
  title = {onóna: Twin Embeddings with Tied Unembeddings — arXiv:2611.10556},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.10556},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bca271cced59; sha256 ff96172375eaf83420fd49cba8832d48a354ceb93ff709c9ed73718d09285ae5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.10556. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Weight tying reduces the parameter cost of language models, but the usual tied output head is limited to a single softmax. We introduce onóna, a two-table parameterization that increases output expressiveness while preserving input-output weight sharing. The Transformer takes the average of two embedding tables as input, and the output head combines the softmax distributions defined by those same tables. A gate conditioned only on the current input token determines the mixture weight. Its values can be precomputed for inference, and the Transformer body remains unchanged. We evaluate onóna across Dense, Gated DeltaNet hybrid, mixture-of-experts, and Attention Residual architectures at 142M, 330M, and 1B scales. Under matched token budgets, onóna lowers final validation loss and external perplexity in 11 of 12 comparisons and raises the average accuracy over eight downstream tasks in 10 of 12. The one exception on loss is the 1B MoE model. The cost is one extra embedding table and one extra vocabulary projection.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.10556 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:52.425Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).