Metraipolis › archive › dc_2ea044e3d418

NE-BERT: A Multilingual Language Model for Nine Northeast Indian Languages — arXiv:2608.18094

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.18094
License (as archived): CC0-1.0
Archived: 2026-10-18T13:44:46.892Z
SHA-256 of the archived text: 69ad39ec3879fcbeadc90b1dc20eddc2eed21e07d509a35317742eeee83981a9
Stable URL of this page: https://metraipolis.com/documento/dc_2ea044e3d418

How to cite

NE-BERT: A Multilingual Language Model for Nine Northeast Indian Languages — arXiv:2608.18094 — arXiv (en), https://arxiv.org/abs/2608.18094 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2ea044e3d418 (Metraipolis archive dc_2ea044e3d418, sha256 69ad39ec3879fcbeadc90b1dc20eddc2eed21e07d509a35317742eeee83981a9)

BibTeX

@misc{dc2ea044e3d418,
  title = {NE-BERT: A Multilingual Language Model for Nine Northeast Indian Languages — arXiv:2608.18094},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.18094},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2ea044e3d418; sha256 69ad39ec3879fcbeadc90b1dc20eddc2eed21e07d509a35317742eeee83981a9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18094. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large pretrained language models have demonstrated remarkable capabilities across diverse languages, yet critically underrepresented low-resource languages remain marginalized. We present NE-BERT, a domain-specific multilingual encoder model trained on approximately 8.3 million sentences spanning 9 Northeast Indian languages and 2 anchor languages (Hindi, English), a linguistically diverse region with minimal representation in existing multilingual models. By employing weighted data sampling and a custom SentencePiece Unigram tokenizer, NE-BERT outperforms IndicBERT-V2 and MuRIL across all 9 Northeast Indian languages, achieving 15.97X and 7.64X lower average perplexity respectively, with 1.50X better tokenization fertility than mBERT. We address critical vocabulary fragmentation issues in extremely low-resource languages such as Pnar (1,002 sentences) and Kokborok (2,463 sentences) through aggressive upsampling strategies. Downstream evaluation on part-of-speech tagging validates practical utility on three Northeast Indian languages. We release NE-BERT, test sets, and training corpus under CC-BY-4.0 to support NLP research and digital inclusion for Northeast Indian communities.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.18094 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:41:17.900Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).