NE-BERT: A Multilingual Language Model for Nine Northeast Indian Languages

# A Multilingual Language Model for Nine Northeast Indian Languages
Source: [https://arxiv.org/html/2608.18094](https://arxiv.org/html/2608.18094)
###### Abstract

Large pretrained language models have demonstrated remarkable capabilities across diverse languages, yet critically underrepresented low\-resource languages remain marginalized\. We presentNE\-BERT, a domain\-specific multilingual encoder model trained on approximately 8\.3 million sentences spanning 9 Northeast Indian languages and 2 anchor languages \(Hindi, English\), a linguistically diverse region with minimal representation in existing multilingual models\. By employing weighted data sampling and a custom SentencePiece Unigram tokenizer, NE\-BERT outperforms IndicBERT\-V2 and MuRIL across all 9 Northeast Indian languages, achieving15\.97×15\.97\\timesand7\.64×7\.64\\timeslower average perplexity respectively, with1\.50×1\.50\\timesbetter tokenization fertility than mBERT\. We address critical vocabulary fragmentation issues in extremely low\-resource languages such as Pnar \(1,002 sentences\) and Kokborok \(2,463 sentences\) through aggressive upsampling strategies\. Downstream evaluation on part\-of\-speech tagging validates practical utility on three Northeast Indian languages\. We release NE\-BERT, test sets, and training corpus under CC\-BY\-4\.0 to support NLP research and digital inclusion for Northeast Indian communities\.

NE\-BERT: A Multilingual Language Model for Nine Northeast Indian Languages

Badal NyalangMWire LabsShillong, Meghalaya, Indianyalang@mwirelabs\.com

## 1Introduction

The performance disparity between high\-resource and low\-resource languages in modern NLP systems reflects and reinforces existing digital inequitiesJoshiet al\.\([2020](https://arxiv.org/html/2608.18094#bib.bib1)\)\. While multilingual models like mBERTDevlinet al\.\([2019](https://arxiv.org/html/2608.18094#bib.bib4)\)provide broad language coverage, they perform poorly on languages with limited web presence and complex morphological structuresLauscheret al\.\([2020](https://arxiv.org/html/2608.18094#bib.bib7)\)\. This gap is particularly pronounced for the indigenous languages of Northeast India, a region home to over 200 distinct languagesMoseley \([2010](https://arxiv.org/html/2608.18094#bib.bib8)\)yet largely absent from mainstream NLP research\.

Northeast Indian languages present unique challenges: extreme resource scarcity \(some with fewer than 1,000 digitized sentences\), agglutinative morphology, script diversity \(Latin, Bengali\-Assamese\), and limited standardization\. Existing regional efforts like IndicBERTKakwaniet al\.\([2020](https://arxiv.org/html/2608.18094#bib.bib3)\)focus primarily on scheduled Indian languages with substantial corpora, leaving languages such as Khasi, Garo, Pnar, Mizo, and Kokborok critically underserved\.

We introduceNE\-BERT, a ModernBERT\-basedWarneret al\.\([2025](https://arxiv.org/html/2608.18094#bib.bib9)\)encoder model specifically designed for Northeast Indian languages\. Our contributions include:

- •A curated multilingual corpus of 8\.3M sentences covering 9 indigenous Northeast Indian languages \(Assamese, Garo, Khasi, Meitei, Mizo, Naga, Nyishi, Pnar, Kokborok\) plus 2 anchor languages \(Hindi, English\) with strategic weighted samplingXueet al\.\([2021](https://arxiv.org/html/2608.18094#bib.bib10)\)\.
- •A custom 50,368\-token SentencePiece Unigram tokenizer optimized for morphologically rich and agglutinative languagesKudo and Richardson \([2018](https://arxiv.org/html/2608.18094#bib.bib2)\), achieving1\.50×1\.50\\timesbetter average tokenization efficiency than mBERT\.
- •Comprehensive evaluation on all 9 Northeast Indian languages demonstrating NE\-BERT outperforms IndicBERT\-V2 and MuRIL, with particularly strong gains \(77–15×15\\times\) on ultra\-low\-resource languages like Pnar, Kokborok, and Nyishi\.

## 2Related Work

### 2\.1Multilingual Language Models

Early multilingual models like mBERTDevlinet al\.\([2019](https://arxiv.org/html/2608.18094#bib.bib4)\)demonstrated cross\-lingual transfer capabilities but suffered from the “curse of multilinguality”Conneauet al\.\([2020](https://arxiv.org/html/2608.18094#bib.bib6)\), performance degradation as language count increases\. XLM\-RoBERTaConneauet al\.\([2020](https://arxiv.org/html/2608.18094#bib.bib6)\)addressed this through larger training corpora \(2\.5TB\) but still exhibited vocabulary fragmentation for low\-resource languages\. Recent work on language\-specific adaptationsRustet al\.\([2021](https://arxiv.org/html/2608.18094#bib.bib11)\)and targeted continued pretrainingChau and Lin \([2020](https://arxiv.org/html/2608.18094#bib.bib12)\)shows promising results for bridging this gap\.

### 2\.2Regional Language Models

Several regional initiatives have emerged to address local language needs\. IndicBERTKakwaniet al\.\([2020](https://arxiv.org/html/2608.18094#bib.bib3)\)covers 12 scheduled Indian languages with 9B tokens, achieving strong performance on Indo\-Aryan and Dravidian languages but with limited coverage of Northeast Indian languages\. Similar efforts for African languagesOguejiet al\.\([2021](https://arxiv.org/html/2608.18094#bib.bib13)\); Alabiet al\.\([2022](https://arxiv.org/html/2608.18094#bib.bib14)\)demonstrate the viability of region\-specific models\. However, these approaches typically focus on languages with substantial existing corpora \(\>\>1M sentences\), leaving ultra\-low\-resource languages unaddressed\.

### 2\.3Tokenization for Low\-Resource Languages

Tokenizer design critically impacts low\-resource language performanceÁcs \([2021](https://arxiv.org/html/2608.18094#bib.bib15)\)\. Byte\-Pair Encoding \(BPE\)Sennrichet al\.\([2016](https://arxiv.org/html/2608.18094#bib.bib16)\), while popular, can fragment morphologically rich words into suboptimal units\. SentencePiece UnigramKudo and Richardson \([2018](https://arxiv.org/html/2608.18094#bib.bib2)\)preserves linguistic structures better for agglutinative languages\. Weighted sampling during tokenizer trainingLample and Conneau \([2019](https://arxiv.org/html/2608.18094#bib.bib17)\)helps balance vocabulary allocation across languages with disparate corpus sizes, critical for our extremely imbalanced dataset\.

## 3Dataset Construction

### 3\.1Language Selection and Sources

We curate data for 9 Northeast Indian languages plus 2 anchor languages \(Table[1](https://arxiv.org/html/2608.18094#S3.T1)\)\. The Northeast Indian languages span three major language families: Sino\-Tibetan \(Meitei, Mizo, Garo, Kokborok, Nyishi, Naga\), Austroasiatic \(Khasi, Pnar\), and Indo\-Aryan \(Assamese\)\. We include Hindi and English as anchor languages to facilitate cross\-lingual transferArtetxeet al\.\([2020](https://arxiv.org/html/2608.18094#bib.bib18)\), particularly for tasks requiring code\-switching support\.

LanguageISOSentencesTokensVirtual CountWeightFamilySourceAnchor LanguagesHindihin3,404,007—170,2000\.05×0\.05\\timesIndo\-AryanHF DatasetsEnglisheng500,000—100,0000\.2×0\.2\\timesGermanicHF DatasetsNortheast Indian LanguagesMeiteimni1,354,32342,504,1811,354,3231\.0×1\.0\\timesSino\-TibetanCuratedAssameseasm1,000,00038,652,3911,000,0001\.0×1\.0\\timesIndo\-AryanCuratedKhasikha1,000,00017,472,6061,000,0001\.0×1\.0\\timesAustroasiaticCuratedMizolus1,000,00026,774,1641,000,0001\.0×1\.0\\timesSino\-TibetanCuratedNyishinjz55,870560,3741,117,40020\.0×20\.0\\timesSino\-TibetanWMT 2025Naganag13,918508,980278,36020\.0×20\.0\\timesSino\-TibetanCuratedGarogrt10,817243,251216,34020\.0×20\.0\\timesSino\-TibetanCuratedKokboroktrp2,46389,851246,300100\.0×100\.0\\timesSino\-TibetanWMT 2025Pnarpbv1,00252,144100,200100\.0×100\.0\\timesAustroasiaticCuratedNE Total4,438,393126,857,942Overall Total8,342,400—6,583,123Table 1:Corpus statistics showing sentence counts, token counts for NE languages, virtual counts after weighted sampling for tokenizer training, language families, and data sources\.Data sources include:

- •Curated Corpora: Meitei, Assamese, Mizo, Khasi, Garo, Pnar, and Naga datasets compiled from government documents, news archives, educational materials, and cultural texts\.
- •WMT 2025 Shared TaskWMT 2025 Organizers \([2025](https://arxiv.org/html/2608.18094#bib.bib29)\): Nyishi and Kokborok parallel corpora from the Workshop on Machine Translation low\-resource language track\.
- •Public Datasets: Hindi from verified Hugging Face datasets; English from standard corpora\.

### 3\.2Data Preprocessing

We apply a rigorous cleaning pipeline to ensure data quality:

1. 1\.Length Filtering: Remove sentences with character length<<20 to eliminate noise, incomplete fragments, and non\-linguistic content\.
2. 2\.Unicode Normalization: Apply NFKC normalizationThe Unicode Consortium \([2021](https://arxiv.org/html/2608.18094#bib.bib19)\)to handle script variations, diacritical marks, and ensure consistency across diverse sources\.
3. 3\.Whitespace Condensation: Collapse multiple spaces and normalize line breaks to standardize formatting\.

We split data into 99\.5% training and 0\.5% validation sets \(random seed 42\)\. A separate held\-out test set is used for final evaluation \(Section[7](https://arxiv.org/html/2608.18094#S7)\)\.

Data Availability: We publicly release our training corpus atBadnyal/ne\-multilingual\-corpusand evaluation test sets atMWirelabs/northeast\-languages\-test\-seton Hugging Face under CC\-BY\-4\.0 license to support reproducibility and future research on Northeast Indian languages\.

### 3\.3Weighted Sampling Strategy

FollowingXueet al\.\([2021](https://arxiv.org/html/2608.18094#bib.bib10)\), we implement aggressive weighted sampling to address extreme resource imbalance\. Table[1](https://arxiv.org/html/2608.18094#S3.T1)shows our weighting scheme: ultra\-low\-resource languages \(Pnar with 1,002 sentences, Kokborok with 2,463 sentences\) receive100×100\\timesupsampling during tokenizer training to ensure adequate vocabulary representation\. This prevents vocabulary starvation where rare languages get fragmented into character\-level tokensRustet al\.\([2021](https://arxiv.org/html/2608.18094#bib.bib11)\), which would severely degrade inference efficiency and model performance\.

The virtual counts in Table[1](https://arxiv.org/html/2608.18094#S3.T1)apply only to tokenizer training; actual MLM training uses raw sentence counts to avoid overfitting on limited data\. Anchor languages are downweighted \(Hindi0\.05×0\.05\\times, English0\.2×0\.2\\times\) to prioritize Northeast language vocabulary while maintaining cross\-lingual transfer capabilities\.

## 4Tokenization

### 4\.1Algorithm Selection

We adopt SentencePiece UnigramKudo and Richardson \([2018](https://arxiv.org/html/2608.18094#bib.bib2)\)over the more common Byte\-Pair Encoding \(BPE\) for two primary reasons:

1. 1\.Linguistic Preservation: Unigram’s probabilistic approach reduces harmful subword fragmentation in morphologically rich and agglutinative languages \(Kokborok, Garo, Meitei\) compared to BPE’s greedy merging strategy\. This is critical for languages where single words can encode complex grammatical information\.
2. 2\.Vocabulary Efficiency: Unigram naturally balances frequent subword allocation across languages without explicit vocabulary partitioning, allowing our weighted sampling strategy to directly influence token boundaries\.

### 4\.2Tokenizer Configuration

Our tokenizer uses the following configuration:

- •Vocabulary Size: 50,368 tokens \(nearest 128\-multiple for efficient Tensor Core execution on modern GPUs\)
- •Character Coverage: 1\.0 \(full Unicode range to handle all scripts\)
- •Maximum Piece Length: 16 characters
- •Shrinking Factor: 0\.75
- •Sub\-iterations: 2
- •Special Tokens:<cls\>\(0\),<pad\>\(1\),<eos\>\(2\),<unk\>\(3\),<mask\>\(4\)

Training on weighted virtual counts \(Table[1](https://arxiv.org/html/2608.18094#S3.T1)\) ensures that common words in Pnar and Kokborok form single tokens rather than fragmenting into multi\-token sequences\. This dramatically reduces inference costs and improves semantic coherence for ultra\-low\-resource languages\.

## 5Model Architecture

We adopt ModernBERT\-baseWarneret al\.\([2025](https://arxiv.org/html/2608.18094#bib.bib9)\)as our foundation due to its architectural improvements over classical BERT:

### 5\.1Architecture Details

- •Encoder Layers: 22 transformer layers
- •Hidden Dimension: 768
- •Attention Heads: 12
- •Total Parameters: 149M - –Embedding layer: 38\.7M parameters - –Encoder layers: 110\.3M parameters
- •Positional Encoding: Rotary Position Embeddings \(RoPE\) withθglobal=160,000\\theta\_\{\\text\{global\}\}=160\{,\}000,θlocal=10,000\\theta\_\{\\text\{local\}\}=10\{,\}000Suet al\.\([2024](https://arxiv.org/html/2608.18094#bib.bib20)\)
- •Attention Mechanism: Flash Attention 2Dao \([2023](https://arxiv.org/html/2608.18094#bib.bib21)\)for memory efficiency
- •Optimization: Unpadding enabled for approximately 30% throughput improvement during training

ModernBERT’s design enables efficient training on longer contexts while maintaining competitive parameter counts relative to BERT\-base \(110M\) and IndicBERT \(66M\)\. The RoPE positional encodings provide better length extrapolation than learned position embeddings, which is beneficial for languages with variable word lengths\.

Model Size Justification: We adopt the 149M parameter configuration as an optimal balance between capability and computational efficiency\. This size is comparable to mBERT \(110M\) while being substantially smaller than IndicBERT\-V2 \(237M\) and MuRIL \(236M\), enabling cost\-effective training \($7\.31 on a single A40 GPU\) and efficient deployment\. Our results demonstrate that appropriate tokenization and targeted training data are more critical than raw parameter count for ultra\-low\-resource language performance\.

## 6Training

### 6\.1Training Configuration

We train NE\-BERT using masked language modeling \(MLM\) with 15% masking probabilityDevlinet al\.\([2019](https://arxiv.org/html/2608.18094#bib.bib4)\)\. We employ dynamic masking where each epoch sees different masked positions, improving generalization compared to static maskingLiuet al\.\([2019](https://arxiv.org/html/2608.18094#bib.bib22)\)\.

Hyperparameters:

- •Batch size: 32 per device with 32 gradient accumulation steps \(effective batch size 1,024\)
- •Learning rate:5×10−45\\times 10^\{\-4\}with cosine decay schedule
- •Warmup steps: 1,500
- •Weight decay: 0\.01
- •Training epochs: 10
- •Precision: Mixed FP16 with TF32 enabled
- •Optimizer: AdamW \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,ϵ=10−8\\epsilon=10^\{\-8\}\)

### 6\.2Compute Infrastructure

Training was conducted on a single NVIDIA A40 GPU \(48GB VRAM\) for approximately 17 hours, with a total compute cost of $7\.31\. This demonstrates the cost\-effectiveness of our approach for resource\-constrained research settings\. We use PyTorch 2\.4\+ with Hugging Face Transformers 4\.48\+ and Flash Attention 2\.x\.

### 6\.3Training Dynamics

Training loss decreased smoothly from approximately 10\.0 at initialization to 1\.62 \(training\) and 1\.64 \(validation\) at convergence over 10 epochs\. The close tracking between training and validation loss indicates no overfitting despite the small corpus size for some languages\. This suggests our weighted sampling strategy and data augmentation through dynamic masking effectively prevent memorization\.

## 7Evaluation

### 7\.1Evaluation Protocol

We evaluate using perplexity \(PPL\) on a held\-out test set of 500 sentences per language\. Perplexity is computed as:

PPL=exp⁡\(1N​∑i=1NℒMLM​\(xi\)\)\\text\{PPL\}=\\exp\\left\(\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathcal\{L\}\_\{\\text\{MLM\}\}\(x\_\{i\}\)\\right\)\(1\)whereℒMLM\\mathcal\{L\}\_\{\\text\{MLM\}\}is the masked language modeling loss andNNis the number of test examples\. Lower perplexity indicates better predictive performance\.

We also measure tokenization fertility, the average number of subword tokens per word, to assess vocabulary efficiencyRustet al\.\([2021](https://arxiv.org/html/2608.18094#bib.bib11)\)\. Lower fertility indicates more efficient tokenization, reducing inference costs and improving semantic coherence\.

Test Set Construction: We construct held\-out test sets of 500 sentences per language through careful deduplication against our training corpus \(Badnyal/ne\-multilingual\-corpus\)\. Test sentences are extracted from newer data sources not present in the training set and filtered to ensure minimum length of 15 characters\. This provides statistically robust perplexity evaluation while avoiding data leakage\. We release test sets publicly atMWirelabs/northeast\-languages\-test\-set\.

Evaluation Coverage: All 9 Northeast Indian languages included in training are evaluated using the constructed test sets\. While Nyishi and Naga test sets are smaller \(extracted from WMT 2025 parallel corpora\), they provide initial validation of model performance across the complete language coverage\.

### 7\.2Baselines

We compare against three widely\-used multilingual models:

- •IndicBERT\-V2Doddapaneniet al\.\([2023](https://arxiv.org/html/2608.18094#bib.bib24)\): 237M parameter encoder trained on 22 Indic languages with 120B tokens\. Enhanced version with expanded language coverage and improved architecture\.
- •MuRILKhanujaet al\.\([2021](https://arxiv.org/html/2608.18094#bib.bib25)\): 236M parameter encoder optimized for Indian languages with 16B tokens\. Google’s multilingual model for Indic NLP\.
- •mBERTDevlinet al\.\([2019](https://arxiv.org/html/2608.18094#bib.bib4)\): 110M parameter encoder covering 104 languages with large\-scale Wikipedia data\. Serves as a general\-purpose multilingual baseline\.

Table[2](https://arxiv.org/html/2608.18094#S7.T2)compares architectural details across all evaluated models\.

Table 2:Model architecture comparison showing parameter counts, layer depth, hidden dimension, and attention heads\.
### 7\.3Results

Table[3](https://arxiv.org/html/2608.18094#S7.T3)presents per\-language perplexity scores across all 9 Northeast Indian languages plus 2 anchor languages\. NE\-BERT achieves the lowest average perplexity across Northeast Indian languages \(2\.21\) compared to IndicBERT\-V2 \(35\.29\), MuRIL \(16\.88\), and mBERT \(2\.77\)\. Performance patterns vary by language resource level: NE\-BERT achieves superior results on ultra\-low\-resource languages \(Pnar, Kokborok, Garo, Nyishi\) where IndicBERT\-V2 and MuRIL exhibit catastrophic performance degradation \(perplexity\>\>60 for Pnar and Nyishi\)\. On higher\-resource languages with extensive Wikipedia coverage \(Assamese, Meitei\), mBERT maintains competitive performance due to its massive pretraining corpus\.

Table 3:Perplexity on 500\-sentence test sets\. NE\-BERT achieves lowest average across all 9 NE languages\.Table[4](https://arxiv.org/html/2608.18094#S7.T4)shows tokenization fertility scores\. NE\-BERT’s custom tokenizer achieves significantly lower average fertility than all baselines across Northeast Indian languages \(1\.68 avg\. vs\. 2\.08 for IndicBERT\-V2, 2\.14 for MuRIL, and 2\.51 for mBERT\)\. Table[5](https://arxiv.org/html/2608.18094#S7.T5)presents bits per character \(BPC\), an alternative compression metric\. NE\-BERT achieves 0\.347 average BPC compared to 1\.497 for IndicBERT\-V2, 1\.271 for MuRIL, and 0\.590 for mBERT, demonstrating superior encoding efficiency\.

Table 4:Tokenization fertility \(tokens per word\) across all 9 NE languages\. Lower values indicate more efficient tokenization\. NE\-BERT achieves lowest average fertility across NE languages\.Table 5:Bits per character \(BPC\) across all 9 NE languages\. Lower is better\. NE\-BERT achieves lowest average BPC\.
### 7\.4Analysis

The performance differences between NE\-BERT and baselines reveal several key insights:

Domain\-Specific Training Advantage: NE\-BERT’s consistent superiority over IndicBERT\-V2 and MuRIL \(average perplexity improvements of15\.97×15\.97\\timesand7\.64×7\.64\\timesrespectively\) validates our hypothesis that domain\-specific models with appropriate tokenization outperform general regional models\. IndicBERT\-V2’s corpus focuses heavily on scheduled languages \(Hindi, Bengali, Tamil, Telugu\) with minimal Northeast representation, resulting in catastrophic failures on ultra\-low\-resource languages \(187\.20 PPL on Nyishi, 66\.92 on Pnar\)\. MuRIL, despite similar parameter count \(236M\), shows severe degradation on non\-scheduled Northeast languages, highlighting the importance of training data composition over model size alone\.

Vocabulary Optimization: The tokenization fertility results \(Table[4](https://arxiv.org/html/2608.18094#S7.T4)\) and BPC scores \(Table[5](https://arxiv.org/html/2608.18094#S7.T5)\) demonstrate the effectiveness of weighted Unigram sampling\. NE\-BERT achieves 1\.68 average tokens/word on Northeast Indian languages versus IndicBERT\-V2’s 2\.08 and MuRIL’s 2\.14, representing 19–21% reduction in sequence length\. The BPC results provide complementary evidence: NE\-BERT’s 0\.347 average BPC demonstrates superior compression compared to IndicBERT\-V2 \(1\.497\) and MuRIL \(1\.271\)\. This efficiency directly translates to faster inference and reduced computational costs, critical factors for deployment in resource\-constrained environments\.

Resource\-Dependent Performance: The results reveal clear patterns based on language resource levels\. On high\-resource languages with extensive Wikipedia coverage \(Assamese: 1M sentences, Meitei: 1\.35M sentences\), mBERT’s massive pretraining corpus \(2\.5TB\) provides competitive performance \(1\.65 and 1\.44 PPL respectively\)\. However, on ultra\-low\-resource languages \(Pnar: 1,002 sentences, Kokborok: 2,463 sentences, Nyishi: 55,870 sentences\) where mBERT has minimal exposure, NE\-BERT’s targeted training yields substantial gains\. The failures of IndicBERT\-V2 and MuRIL on these languages \(66\.92–187\.20 PPL\) demonstrate that simply scaling model size without adequate language representation is insufficient for ultra\-low\-resource scenarios\.

Script Diversity Handling: The fertility and BPC improvements are particularly striking for non\-Latin scripts\. Assamese \(Bengali\-Assamese script\) shows substantial efficiency gains in BPC \(0\.230 vs\. 0\.442 for mBERT, 0\.880 for IndicBERT\-V2\), while Meitei \(also using Bengali\-Assamese script\) demonstrates similar patterns\. For Latin\-script languages, NE\-BERT achieves competitive or superior efficiency across all metrics\. This validates our choice of Unigram tokenization with weighted sampling, which better preserves script\-specific morphological boundaries than BPE\-based approaches used in baseline models\.

Anchor Language Transfer: The strong performance on both anchor languages \(English: 1\.55 PPL, Hindi: 1\.43 PPL\) despite being downweighted during tokenizer training \(0\.2× and 0\.05× respectively\) demonstrates effective cross\-lingual transfer\. NE\-BERT outperforms IndicBERT\-V2 \(14\.81 and 10\.08 PPL\) and MuRIL \(5\.57 and 5\.75 PPL\) on both anchor languages, suggesting that our weighted sampling strategy successfully balances vocabulary allocation without sacrificing anchor language performance\. This is particularly important for real\-world deployment where code\-switching between Northeast languages and Hindi/English is common\.

## 8Downstream Evaluation

To validate practical utility beyond perplexity metrics, we evaluate NE\-BERT on part\-of\-speech \(POS\) tagging across three Northeast Indian languages using Universal Dependencies annotationsNivreet al\.\([2020](https://arxiv.org/html/2608.18094#bib.bib26)\)\.

### 8\.1Experimental Setup

We fine\-tune all models on language\-specific POS tagging using the following datasets:

- •Khasi: 519 sentences \(414 train, 52 dev, 53 test\)Ghoshet al\.\([2025](https://arxiv.org/html/2608.18094#bib.bib27)\)
- •Mizo: 502 sentences \(402 train, 50 dev, 50 test\)Ghoshet al\.\([2025](https://arxiv.org/html/2608.18094#bib.bib27)\)
- •Nagamese: 214 sentences \(170 train, 22 dev, 22 test\)Maitiet al\.\([2025](https://arxiv.org/html/2608.18094#bib.bib28)\)

All models are fine\-tuned for 5 epochs with learning rate2×10−52\\times 10^\{\-5\}, batch size 16, and standard cross\-entropy loss\. We compare NE\-BERT against mBERT, IndicBERT\-V2Doddapaneniet al\.\([2023](https://arxiv.org/html/2608.18094#bib.bib24)\), and MuRILKhanujaet al\.\([2021](https://arxiv.org/html/2608.18094#bib.bib25)\)\. We emphasize that these results are illustrative rather than definitive, given the small size of available annotated datasets\.

### 8\.2POS Tagging Performance

Table[6](https://arxiv.org/html/2608.18094#S8.T6)presents POS tagging accuracy on test sets\. NE\-BERT achieves the highest accuracy across all three languages, with an average of 82\.4%, outperforming mBERT by 9\.1 percentage points and IndicBERT\-V2 by 23\.2 percentage points\.

Table 6:POS tagging accuracy \(%\) on test sets\. NE\-BERT outperforms all baselines across all languages\.The results demonstrate that NE\-BERT’s specialized vocabulary and targeted pretraining translate to improved performance on downstream tasks\. The particularly large gap against IndicBERT\-V2 and MuRIL on Nagamese \(86\.4% vs\. 41\.7–44\.8%\) highlights the importance of adequate language representation during pretraining\.

## 9Conclusion

We present NE\-BERT, a multilingual encoder model for 9 Northeast Indian languages, demonstrating that domain\-specific models with appropriate tokenization can effectively serve ultra\-low\-resource languages with as few as 1,000 training sentences\. Our model outperforms IndicBERT\-V2 across all 9 evaluated languages \(15\.97×15\.97\\timesaverage improvement\) and achieves competitive or superior performance compared to mBERT \(2\.21 vs\. 2\.76 average PPL\), with particularly strong gains on ultra\-low\-resource languages like Pnar \(66\.92 vs\. 2\.92 PPL\), Kokborok, and Nyishi \(187\.20 vs\. 4\.33 PPL\)\. Downstream evaluation on POS tagging validates practical utility, with NE\-BERT achieving 82\.4% average accuracy across Khasi, Mizo, and Nagamese, outperforming mBERT by 9\.1 percentage points\.

The key innovations \(weighted Unigram tokenization, aggressive upsampling for ultra\-low\-resource languages, and cost\-effective training at $7\.31 on a single A40 GPU\) provide a practical blueprint for developing language models for underrepresented languages worldwide\. Our tokenization efficiency improvements \(1\.50× better fertility than mBERT, 4\.3× better BPC than IndicBERT\-V2\) demonstrate that careful vocabulary optimization can substantially reduce inference costs while improving model quality\.

This work represents a foundation for future NLP research on Northeast Indian languages\. We release NE\-BERT, tokenizer, training code, and documentation under CC\-BY\-4\.0 to support community\-driven improvements and applications\.

## Limitations

Limited Downstream Evaluation: While we validate NE\-BERT on part\-of\-speech tagging for three languages \(Khasi, Mizo, Nagamese\), comprehensive evaluation across diverse tasks \(named entity recognition, sentiment analysis, machine translation\) and all nine trained languages remains future work\. The small scale of available POS datasets \(214–519 sentences\) limits statistical robustness of downstream results\.

Encoder\-Only Architecture: NE\-BERT is limited to representation tasks \(classification, NER, embedding generation\)\. Generation tasks \(machine translation, summarization, dialogue\) require decoder or encoder\-decoder architectures\.

Ultra\-Low\-Resource Vulnerability: Languages with fewer than 3,000 sentences \(Pnar, Kokborok, Garo, Naga\) remain vulnerable to distribution shift\. While weighted sampling mitigates vocabulary fragmentation, these models may exhibit unexpected behavior on out\-of\-distribution inputs\.

## Future Work

Comprehensive Downstream Evaluation: We are developing benchmark datasets for named entity recognition, sentiment analysis, and additional part\-of\-speech tagging datasets across all 9 Northeast Indian languages, including Nyishi and Naga\. Expanding POS evaluation beyond the current three languages and increasing dataset sizes will provide more robust assessment of NE\-BERT’s practical utility\.

Decoder Models: Extending our approach to autoregressive architectures would enable generation tasks\. We plan to train decoder\-only models using the same data curation and tokenization strategies, targeting conversational assistants for Northeast Indian languages\.

Data Expansion: Active collaboration with native speaker communities and linguistic experts to expand corpora, particularly for ultra\-low\-resource languages\. Target is 10K\+ sentences for Pnar, Kokborok, Garo, and Naga\.

Cross\-Lingual Transfer Studies: Systematic investigation of zero\-shot and few\-shot transfer capabilities to related but unrepresented languages \(e\.g\., Bodo, Karbi, Dimasa\) to assess generalization beyond training languages\.

Deployment Studies: Real\-world deployment pilots with government and educational institutions to assess model performance on authentic tasks and gather community feedback\.

## Ethical Considerations

### Bias and Representation

Language models inherit biases present in training dataBlodgettet al\.\([2020](https://arxiv.org/html/2608.18094#bib.bib23)\)\. Our collected corpora, derived from public government, educational, and media sources may contain gender, religious, caste, and other social biases reflecting the perspectives of text authors and publishers\. Ultra\-low\-resource languages face additional risks:

- •Dominance Bias: High\-resource languages \(Meitei, Assamese\) may dominate model behavior despite weighted sampling, potentially marginalizing ultra\-low\-resource languages in multilingual contexts\.
- •Quality Variance: Limited data for Pnar, Kokborok, Garo, and Naga increases sensitivity to data quality issues and potential amplification of biases present in small corpora\.
- •Hallucination Risk: Models may generate plausible\-sounding but incorrect content when faced with out\-of\-distribution inputs for ultra\-low\-resource languages\.

We recommend thorough evaluation and community feedback before deploying NE\-BERT in sensitive applications such as education, government services, or content moderation\.

### Linguistic and Cultural Impact

Language technologies can both preserve and threaten linguistic diversityBird \([2020](https://arxiv.org/html/2608.18094#bib.bib5)\)\. While NE\-BERT enables digital inclusion for marginalized languages, potential negative impacts include:

- •Standardization Pressure: Models may favor formal or written registers over spoken varieties, potentially marginalizing dialectal variation and informal language use\.
- •Power Dynamics: Deployment without community consent or benefit\-sharing could reinforce extractive relationships between researchers and language communities\.
- •Representation Gaps: Our dataset primarily reflects government and educational registers, potentially underrepresenting oral traditions, youth language, and non\-elite perspectives\.

We are committed to:

- •Transparent documentation of data sources, model limitations, and intended use cases
- •Ongoing collaboration with native speaker communities for feedback and validation
- •Benefit\-sharing through open\-source release and support for community\-driven applications
- •Respect for community decisions regarding data use and model deployment

## Acknowledgements

We thank the reviewers for their valuable feedback\. We are grateful to the Workshop on Machine Translation \(WMT\) 2025 organizersWMT 2025 Organizers \([2025](https://arxiv.org/html/2608.18094#bib.bib29)\)for providing the Nyishi and Kokborok parallel corpora\. We extend our gratitude to the native speaker communities of Northeast India for their contributions to language preservation efforts\. This work was supported by MWire Labs\.

## References

- J\. Ács \(2021\)Evaluating multilingual text encoders for unsupervised cross\-lingual retrieval\.InAdvances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021,pp\. 342–349\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-72113-8%5F23)Cited by:[§2\.3](https://arxiv.org/html/2608.18094#S2.SS3.p1.1)\.
- J\. O\. Alabi, D\. I\. Adelani, M\. Mosbach, and D\. Klakow \(2022\)Adapting pre\-trained language models to African languages via multilingual adaptive fine\-tuning\.InProceedings of the 29th International Conference on Computational Linguistics,pp\. 4336–4349\.External Links:[Link](https://aclanthology.org/2022.coling-1.382)Cited by:[§2\.2](https://arxiv.org/html/2608.18094#S2.SS2.p1.1)\.
- M\. Artetxe, S\. Ruder, and D\. Yogatama \(2020\)On the cross\-lingual transferability of monolingual representations\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4623–4637\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.421)Cited by:[§3\.1](https://arxiv.org/html/2608.18094#S3.SS1.p1.1)\.
- S\. Bird \(2020\)Decolonising speech and language technology\.InProceedings of the 28th International Conference on Computational Linguistics,pp\. 3504–3519\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.coling-main.313)Cited by:[Linguistic and Cultural Impact](https://arxiv.org/html/2608.18094#Sx3.SSx2.p1.1)\.
- S\. L\. Blodgett, S\. Barocas, H\. Daumé III, and H\. Wallach \(2020\)Language \(technology\) is power: a critical survey of “bias” in NLP\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 5454–5476\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.485)Cited by:[Bias and Representation](https://arxiv.org/html/2608.18094#Sx3.SSx1.p1.1)\.
- E\. C\. Chau and L\. H\. Lin \(2020\)Parsing with multilingual BERT, a small corpus, and a small treebank\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 1324–1334\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.118)Cited by:[§2\.1](https://arxiv.org/html/2608.18094#S2.SS1.p1.1)\.
- A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov \(2020\)Unsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 8440–8451\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747)Cited by:[§2\.1](https://arxiv.org/html/2608.18094#S2.SS1.p1.1)\.
- T\. Dao \(2023\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.arXiv preprint arXiv:2307\.08691\.External Links:[Link](https://arxiv.org/abs/2307.08691)Cited by:[6th item](https://arxiv.org/html/2608.18094#S5.I1.i6.p1.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4171–4186\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§1](https://arxiv.org/html/2608.18094#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.18094#S2.SS1.p1.1),[§6\.1](https://arxiv.org/html/2608.18094#S6.SS1.p1.1),[3rd item](https://arxiv.org/html/2608.18094#S7.I1.i3.p1.1)\.
- S\. Doddapaneni, R\. Aralikatte, G\. Ramesh, S\. Goyal, M\. M\. Khapra, A\. Kunchukuttan, and P\. Kumar \(2023\)Towards leaving no Indic language behind: building monolingual corpora, benchmark and models for Indic languages\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada,pp\. 12402–12426\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.693)Cited by:[1st item](https://arxiv.org/html/2608.18094#S7.I1.i1.p1.1),[§8\.1](https://arxiv.org/html/2608.18094#S8.SS1.p2.1)\.
- S\. Ghosh, N\. Vuppala, D\. Marbaniang, and H\. Lalsiam \(2025\)Towards resource\-rich Mizo and Khasi in NLP: resource development, synthetic data generation and model building\.InProceedings of the Third Workshop on Language Technology for Equality, Diversity, Inclusion,pp\. 176–185\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.law-1.18)Cited by:[1st item](https://arxiv.org/html/2608.18094#S8.I1.i1.p1.1),[2nd item](https://arxiv.org/html/2608.18094#S8.I1.i2.p1.1)\.
- P\. Joshi, S\. Santy, A\. Budhiraja, K\. Bali, and M\. Choudhury \(2020\)The state and fate of linguistic diversity and inclusion in the NLP world\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 6282–6293\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.560)Cited by:[§1](https://arxiv.org/html/2608.18094#S1.p1.1)\.
- D\. Kakwani, A\. Kunchukuttan, S\. Golla, N\. Gokul, A\. Bhattacharyya, M\. M\. Khapra, and P\. Kumar \(2020\)IndicNLPSuite: monolingual corpora, evaluation benchmarks and pre\-trained multilingual language models for Indian languages\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 4948–4961\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.445)Cited by:[§1](https://arxiv.org/html/2608.18094#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.18094#S2.SS2.p1.1)\.
- S\. Khanuja, D\. Bansal, S\. Mehtani, S\. Khosla, A\. Dey, B\. Gopalan, D\. K\. Margam, P\. Aggarwal, R\. T\. Nagipogu, S\. Dave, S\. Gupta, S\. C\. B\. Gali, V\. Subramanian, and P\. Talukdar \(2021\)MuRIL: multilingual representations for Indian languages\.arXiv preprint arXiv:2103\.10730\.External Links:[Link](https://arxiv.org/abs/2103.10730)Cited by:[2nd item](https://arxiv.org/html/2608.18094#S7.I1.i2.p1.1),[§8\.1](https://arxiv.org/html/2608.18094#S8.SS1.p2.1)\.
- T\. Kudo and J\. Richardson \(2018\)SentencePiece: a simple and language independent approach to subword tokenization and detokenization\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp\. 66–71\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-2012)Cited by:[2nd item](https://arxiv.org/html/2608.18094#S1.I1.i2.p1.1),[§2\.3](https://arxiv.org/html/2608.18094#S2.SS3.p1.1),[§4\.1](https://arxiv.org/html/2608.18094#S4.SS1.p1.1)\.
- G\. Lample and A\. Conneau \(2019\)Cross\-lingual language model pretraining\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\. 7059–7069\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/c04c19c2c2474dbf5f7ac4372c5b9af1-Abstract.html)Cited by:[§2\.3](https://arxiv.org/html/2608.18094#S2.SS3.p1.1)\.
- A\. Lauscher, V\. Ravishankar, I\. Vulić, and G\. Glavaš \(2020\)From zero to hero: on the limitations of zero\-shot language transfer with multilingual transformers\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 4483–4499\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.363)Cited by:[§1](https://arxiv.org/html/2608.18094#S1.p1.1)\.
- Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov \(2019\)RoBERTa: a robustly optimized BERT pretraining approach\.arXiv preprint arXiv:1907\.11692\.External Links:[Link](https://arxiv.org/abs/1907.11692)Cited by:[§6\.1](https://arxiv.org/html/2608.18094#S6.SS1.p1.1)\.
- A\. Maiti, M\. Pandey, and M\. Mandal \(2025\)NagaNLP: bootstrapping NLP for low\-resource Nagamese with closed\-loop synthetic data\.Note:arXiv preprint arXiv:2512\.12537External Links:[Link](https://arxiv.org/abs/2512.12537)Cited by:[3rd item](https://arxiv.org/html/2608.18094#S8.I1.i3.p1.1)\.
- C\. Moseley \(2010\)Atlas of the world’s languages in danger\.3rd edition,UNESCO Publishing\.External Links:[Link](https://unesdoc.unesco.org/ark:/48223/pf0000187026)Cited by:[§1](https://arxiv.org/html/2608.18094#S1.p1.1)\.
- J\. Nivre, M\. de Marneffe, F\. Ginter, J\. Hajič, C\. D\. Manning, S\. Pyysalo, S\. Schuster, F\. Tyers, and D\. Zeman \(2020\)Universal Dependencies v2: an evergrowing multilingual treebank collection\.InProceedings of the Twelfth Language Resources and Evaluation Conference,pp\. 4034–4043\.External Links:[Link](https://aclanthology.org/2020.lrec-1.497)Cited by:[§8](https://arxiv.org/html/2608.18094#S8.p1.1)\.
- K\. Ogueji, Y\. Zhu, and J\. Lin \(2021\)Small data? No problem\! Exploring the viability of pretrained multilingual language models for low\-resourced languages\.InProceedings of the 1st Workshop on Multilingual Representation Learning,pp\. 116–126\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.mrl-1.11)Cited by:[§2\.2](https://arxiv.org/html/2608.18094#S2.SS2.p1.1)\.
- P\. Rust, J\. Pfeiffer, I\. Vulić, S\. Ruder, and I\. Gurevych \(2021\)How good is your tokenizer? On the monolingual performance of multilingual language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 3118–3135\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.243)Cited by:[§2\.1](https://arxiv.org/html/2608.18094#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2608.18094#S3.SS3.p1.1),[§7\.1](https://arxiv.org/html/2608.18094#S7.SS1.p2.1)\.
- R\. Sennrich, B\. Haddow, and A\. Birch \(2016\)Neural machine translation of rare words with subword units\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1715–1725\.External Links:[Document](https://dx.doi.org/10.18653/v1/P16-1162)Cited by:[§2\.3](https://arxiv.org/html/2608.18094#S2.SS3.p1.1)\.
- J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu \(2024\)RoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.External Links:[Document](https://dx.doi.org/10.1016/j.neucom.2023.127063)Cited by:[5th item](https://arxiv.org/html/2608.18094#S5.I1.i5.p1.2)\.
- The Unicode Consortium \(2021\)The Unicode standard, version 14\.0\.Mountain View, CA\.External Links:[Link](https://www.unicode.org/versions/Unicode14.0.0/)Cited by:[item 2](https://arxiv.org/html/2608.18094#S3.I2.i2.p1.1)\.
- B\. Warner, P\. Maruf, M\. Treviso, A\. F\. Aji, I\. Jauregi Unanue, J\. Phang, B\. Shao, J\. Xu, C\. J\. Yee, J\. Lin, C\. Thorne,et al\.\(2025\)Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.127)Cited by:[§1](https://arxiv.org/html/2608.18094#S1.p3.1),[§5](https://arxiv.org/html/2608.18094#S5.p1.1)\.
- WMT 2025 Organizers \(2025\)Findings of the WMT 2025 shared task on low\-resource language translation\.InProceedings of the Tenth Conference on Machine Translation,External Links:[Document](https://dx.doi.org/10.18653/v1/2025.wmt-1.29)Cited by:[2nd item](https://arxiv.org/html/2608.18094#S3.I1.i2.p1.1),[Acknowledgements](https://arxiv.org/html/2608.18094#Sx4.p1.1)\.
- L\. Xue, N\. Constant, A\. Roberts, M\. Kale, R\. Al\-Rfou, A\. Siddhant, A\. Barua, and C\. Raffel \(2021\)MT5: a massively multilingual pre\-trained text\-to\-text transformer\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 483–498\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.41)Cited by:[1st item](https://arxiv.org/html/2608.18094#S1.I1.i1.p1.1),[§3\.3](https://arxiv.org/html/2608.18094#S3.SS3.p1.1)\.

## Appendix

## Appendix ATraining Loss Curves

Figure[1](https://arxiv.org/html/2608.18094#A1.F1)shows the complete training and validation loss curves over 10 epochs\. The close tracking between training and validation loss throughout training indicates effective generalization without overfitting, despite the small corpus size for ultra\-low\-resource languages\.

Steps \(×100\\times 100\)Loss02468100255075100TrainingValidationFigure 1:Training and validation loss curves over 10 epochs\.
## Appendix BLanguage Examples

Table[2](https://arxiv.org/html/2608.18094#A2.F2)presents representative sentences from each of the 9 Northeast Indian languages in our corpus, demonstrating script diversity and morphological variation across language families\.

![Refer to caption](https://arxiv.org/html/2608.18094v1/langaugesample.png)Figure 2:Representative sentences from each Northeast Indian language in our corpus\. For Bengali\-Assamese script languages \(Assamese, Meitei\), both the original script and Latin transliterations \(in italics\) are shown\.