BERT-based Models vs. Large Language Models for Low-Resource Named Entity Recognition: A Comparative Study on Marathi

# BERT-based Models vs. Large Language Models for Low-Resource Named Entity Recognition: A Comparative Study on Marathi
Source: [https://arxiv.org/html/2607.23344](https://arxiv.org/html/2607.23344)
Hariom Ingle1,3Ronit Ghode1,3Ishwari Gondkar1,3Jidnyasa Harad1,3Raviraj Joshi2,3 1Department of Information Technology, PICT, Pune, India 2Indian Institute of Technology Madras, Chennai, India 3L3Cube Labs, Pune, India [ravirajoshi@gmail\.com](https://arxiv.org/html/2607.23344v1/[email protected])

###### Abstract

Named Entity Recognition \(NER\) for low\-resource languages such as Marathi remains a challenging task due to limited annotated resources and linguistic complexity\. Although recent Large Language Models \(LLMs\) have demonstrated strong performance across a wide range of natural language processing tasks, their effectiveness for language\-specific NER in low\-resource settings remains uncertain\. In this study, we fine\-tune MahaBERT\-v2 on different variants of the MahaNER dataset and systematically compare the performance of these models with an existing MahaNER baseline and prominent general\-purpose LLMs, including Gemini, LLaMA\-3\.3\-70B, and Gemma models\. All models are evaluated on a Marathi NER test dataset using standard metrics of precision, recall, and F1\-score\. The experimental results show that the fine\-tuned MahaBERT\-based models consistently outperform both the baseline and all evaluated LLMs, with the fine\-tuned models achieving F1\-scores ranging from 0\.88 to 0\.91, surpassing the existing MahaNER model \(0\.8843\) and significantly exceeding the performance of LLM\-based approaches, whose F1\-scores range from 0\.57 to 0\.69\. These findings demonstrate that task\-specific, language\-focused models trained on domain\-relevant data remain more effective than general\-purpose LLMs for Marathi NER, highlighting the continued importance of specialized architectures for low\-resource language processing\.

BERT\-based Models vs\. Large Language Models for Low\-Resource Named Entity Recognition: A Comparative Study on Marathi

Hariom Ingle1,3Ronit Ghode1,3Ishwari Gondkar1,3Jidnyasa Harad1,3Raviraj Joshi2,31Department of Information Technology, PICT, Pune, India2Indian Institute of Technology Madras, Chennai, India3L3Cube Labs, Pune, India[ravirajoshi@gmail\.com](https://arxiv.org/html/2607.23344v1/[email protected])

## 1Introduction

Named Entity Recognition \(NER\) is a fundamental task in Natural Language Processing \(NLP\) that involves identifying and classifying named entities such as persons, organizations, locations, and other predefined categories within textLiet al\.\([2022](https://arxiv.org/html/2607.23344#bib.bib12)\)\. NER serves as a critical component in downstream applications including information extraction, question answering, machine translation, and knowledge graph construction\. While significant advancements have been achieved for high\-resource languages like English, developing robust NER systems for low\-resource languages remains a persistent challenge\.

Marathi is an Indo\-Aryan language spoken by over 83 million people, primarily in the state of Maharashtra, India\. It is one of the major scheduled languages of India and holds official language status\. Despite its large speaker base, Marathi is considered a low\-resource language in the context of NLPJoshi \([2022b](https://arxiv.org/html/2607.23344#bib.bib16)\)\. This is because it has limited annotated corpora, few pretrained language models, and relatively little NLP tooling compared to languages like English, French, or even Hindi\. Marathi also presents unique linguistic complexities, including rich morphology, inflectional variations, and flexible word order, all of which make tasks like NER harder to solve\.

NER in Marathi is particularly difficult for several interconnected reasons\. First, Marathi exhibits rich morphology, where a single word can take many different forms depending on its grammatical role, which increases vocabulary size and makes it harder for models to generalise\. Second, unlike English, Marathi written in the Devanagari script does not use capital letters, removing a strong signal that English NER models rely on to detect proper nounsSang and Meulder \([2003](https://arxiv.org/html/2607.23344#bib.bib15)\)\. Third, the flexible word order of Marathi allows the subject, verb, and object to appear in varying positions within a sentence, making positional features less reliable for entity detection\. Fourth, Marathi text, especially in digital and social media, frequently mixes words from Hindi and English, introducing additional ambiguity through code\-mixing\. Finally, there are very few large, manually annotated Marathi NER datasets available, which limits the training of data\-hungry deep learning models and compounds all of the above challenges\.

In this work, we address these challenges by fine\-tuning variants of the L3Cube\-MahaBERT v2 modelJoshi \([2022a](https://arxiv.org/html/2607.23344#bib.bib2)\)on the MahaNER datasetLitakeet al\.\([2022](https://arxiv.org/html/2607.23344#bib.bib3)\)under different training configurations\. We specifically study the effect of context length expansion using two data augmentation strategies: self\-concatenation and random concatenation\. In the self\-concatenation approach, each training sentence is repeated multiple times \(4×4\\timesor10×10\\times\) within a single input sequence\. In the random concatenation approach, multiple different sentences are combined into a single longer sequence\. We call the resulting model variants MahaNER\-normal\-repeat\-4x, MahaNER\-normal\-repeat\-10x, MahaNER\-random\-repeat\-4x, and MahaNER\-random\-repeat\-10x\.

We then compare these fine\-tuned models against several general\-purpose Large Language Models evaluated in a zero\-shot setting, including Gemini, LLaMA\-3\.3\-70BDubeyet al\.\([2024](https://arxiv.org/html/2607.23344#bib.bib7)\), and GemmaGemma Teamet al\.\([2024](https://arxiv.org/html/2607.23344#bib.bib8)\)\. The goal is to understand whether task\-specific fine\-tuned models still hold an advantage over powerful general\-purpose LLMs for Marathi NER, and what role data augmentation plays in improving NER performance\.

The main contributions of this paper are:

- •We fine\-tune and evaluate multiple MahaBERT\-v2\-based NER models using different data augmentation strategies\.
- •We compare fine\-tuned Marathi NER models against state\-of\-the\-art LLMs in a zero\-shot setting\.
- •We show that language\-specific fine\-tuned models significantly outperform general\-purpose LLMs for Marathi NER\.
- •We provide a detailed analysis of how training data augmentation affects model performance on different test sets\.

## 2Related Work

### 2\.1Evolution of NER Systems

NER has a long research history spanning rule\-based, statistical, and neural approachesLiet al\.\([2022](https://arxiv.org/html/2607.23344#bib.bib12)\)\. Early NER systems used hand\-written rules and dictionaries\. These systems required a lot of manual effort and did not generalise well to new domains or languages\. Statistical models like Hidden Markov Models \(HMMs\) and Maximum Entropy classifiers then replaced rule\-based systems by learning patterns from annotated data\. Conditional Random Fields \(CRFs\) further improved results by modelling dependencies between output labels, making them well\-suited for sequence labelling tasks like NERLampleet al\.\([2016](https://arxiv.org/html/2607.23344#bib.bib13)\)\. Benchmark datasets such as CoNLL\-2003Sang and Meulder \([2003](https://arxiv.org/html/2607.23344#bib.bib15)\)played a central role in driving progress across these approaches\.

The rise of deep learning brought significant improvements\. Convolutional Neural Networks \(CNNs\) and Long Short\-Term Memory \(LSTM\) networks, especially Bidirectional LSTMs \(BiLSTMs\), were shown to learn useful representations of text directly from data\. These models reduced the need for hand\-crafted features\. Combining BiLSTMs with CRF decoding layers became a popular and strong approach for NERLampleet al\.\([2016](https://arxiv.org/html/2607.23344#bib.bib13)\)\.

The introduction of transformer\-based modelsVaswaniet al\.\([2017](https://arxiv.org/html/2607.23344#bib.bib5)\), starting with BERTDevlinet al\.\([2019](https://arxiv.org/html/2607.23344#bib.bib1)\), fundamentally changed the field\. BERT learns deep contextual representations of words by pretraining on large text corpora using masked language modelling\. Fine\-tuning BERT on NER datasets consistently achieved state\-of\-the\-art results across many languages and benchmarks\. Subsequent models such as RoBERTaLiuet al\.\([2019](https://arxiv.org/html/2607.23344#bib.bib11)\)and T5Raffelet al\.\([2020](https://arxiv.org/html/2607.23344#bib.bib14)\)further advanced pretraining methodology, demonstrating that better training objectives and more data consistently improve downstream task performance including NER\.

### 2\.2Marathi NLP Resources

Marathi NLP has seen growing interest over the past few years\. The L3Cube\-MahaCorpus provided a large Marathi text corpus, and MahaBERT was pretrained on this corpus to create the first Marathi\-specific BERT modelJoshi \([2022a](https://arxiv.org/html/2607.23344#bib.bib2)\)\. This model significantly outperformed multilingual BERT \(mBERT\) on downstream Marathi tasks\.

MahaNERLitakeet al\.\([2022](https://arxiv.org/html/2607.23344#bib.bib3)\)provided the first large, manually annotated Marathi NER dataset\. It defined entity categories including Person, Organisation, and Location, and showed that MahaBERT\-based models outperform multilingual alternatives like mBERT and XLM\-RConneauet al\.\([2020](https://arxiv.org/html/2607.23344#bib.bib6)\)by a clear margin\. Later work introduced social media NER for Marathi through the L3Cube\-MahaSocialNER datasetChaudhariet al\.\([2023](https://arxiv.org/html/2607.23344#bib.bib4)\), which highlighted the challenges of informal text and code\-mixing in low\-resource settings\.

### 2\.3LLMs for NER

Large Language Models such as GPT\-4, LLaMADubeyet al\.\([2024](https://arxiv.org/html/2607.23344#bib.bib7)\), and GemmaGemma Teamet al\.\([2024](https://arxiv.org/html/2607.23344#bib.bib8)\)have shown impressive performance on many NLP tasks in a zero\-shot or few\-shot settingBrownet al\.\([2020](https://arxiv.org/html/2607.23344#bib.bib9)\)\. Chain\-of\-thought promptingWeiet al\.\([2022](https://arxiv.org/html/2607.23344#bib.bib10)\)has further improved LLM reasoning on complex tasks\. However, for structured prediction tasks like NER, which require precise token\-level labelling, LLMs often struggle\. They may produce inconsistent output formats, miss entity boundaries, or fail to follow labelling instructions consistently\. This problem is more pronounced for low\-resource languages where the LLM’s pretraining data is sparse\. Despite these known limitations, a systematic comparison between fine\-tuned Marathi NER models and state\-of\-the\-art LLMs has not been done before\. Our work fills this gap\.

## 3Dataset

### 3\.1MahaNER Corpus

All experiments in this paper are based on the MahaNER corpusLitakeet al\.\([2022](https://arxiv.org/html/2607.23344#bib.bib3)\), which is the primary benchmark dataset for Marathi Named Entity Recognition\. MahaNER was constructed by manually annotating Marathi news text with entity labels following annotation conventions similar to CoNLL\-2003Sang and Meulder \([2003](https://arxiv.org/html/2607.23344#bib.bib15)\)\. The dataset covers standard NER entity types including Person \(PER\), Organisation \(ORG\), and Location \(LOC\), along with a few other categories\. It is the largest and most widely used annotated Marathi NER resource available\.

The corpus was annotated using a non\-IOB tagging scheme, where each token is assigned one of the following label types:

- •TYPE:Token is a named entity of a given type \(e\.g\., PER, ORG, LOC\)\.
- •O:Token is outside any named entity\.

Table[1](https://arxiv.org/html/2607.23344#S3.T1)shows the approximate size and entity distribution of the MahaNER corpus\.

Table 1:Key properties of the MahaNER corpus used in this study\.
### 3\.2Test Dataset Variants

Three test dataset variants were derived from the MahaNER corpus for evaluation\. Table[2](https://arxiv.org/html/2607.23344#S3.T2)summarises each variant along with its construction strategy and objective\.

Table 2:MahaNER\-derived test datasets\. Each variant targets a different aspect of model robustness\.TheNormaldataset is the standard test split of MahaNER with naturally occurring entity distributions and no modifications\. It serves as the primary baseline for comparison\.

TheNormal Repeat 4xdataset is created by repeating each test sentence four times within a single sequence\. This tests whether models can make consistent predictions when the same sentence appears multiple times in a longer input\.

TheRandom Repeat 4xdataset is built by randomly combining four different test sentences into a single input sequence\. This introduces more varied entity patterns within each sequence and tests model robustness to diverse contexts\.

### 3\.3Training Data Augmentation

To study the effect of training data augmentation, multiple augmented training sets were created from the MahaNER training split using the same repetition strategies described above\. Repetition factors of4×4\\timesand10×10\\timeswere used for both normal\-repeat and random\-repeat strategies\. These augmented training sets give the model exposure to longer input sequences during training, which may improve performance on longer test sequences\. Importantly, the augmentation was applied only to the training data; all test sets remain unmodified and in their original form\.

## 4Methodology

This section describes the two types of systems we evaluate: task\-specific fine\-tuned models and zero\-shot Large Language Models\. Figure[1](https://arxiv.org/html/2607.23344#S4.F1)gives an overview of the full evaluation pipeline\.

Marathi Test Sentences \(500\)Fine\-tunedMahaBERT\-v2Zero\-shotLLMsBase, normal\-repeat\-4x,normal\-repeat\-10x,random\-repeat\-4x, random\-repeat\-10xGemini, LLaMA\-3\.3\-70B,Gemma\-2\-9B, Gemma\-2\-27BToken\-level Evaluation\(Precision, Recall, F1\)Figure 1:Overview of the evaluation pipeline\. Both fine\-tuned MahaBERT models and zero\-shot LLMs are tested on the same 500 Marathi sentences and evaluated using the same metrics\.### 4\.1Task\-Specific NER Models

All task\-specific models are based on the L3Cube\-MahaBERT v2 architectureJoshi \([2022a](https://arxiv.org/html/2607.23344#bib.bib2)\), a BERT modelDevlinet al\.\([2019](https://arxiv.org/html/2607.23344#bib.bib1)\)pretrained on a large Marathi monolingual corpus\. Fine\-tuning adds a token classification head on top of the pretrained encoder, which assigns a non\-IOB NER label to each input token\.

Five fine\-tuned model variants were trained using different augmented training sets:

1. 1\.MahaNER Base Model:Trained on the original MahaNER training set without any augmentation\.
2. 2\.MahaNER\-normal\-repeat\-4x:Each training sentence is repeated 4 times in one sequence\. This increases sequence length and repeats the same context\.
3. 3\.MahaNER\-normal\-repeat\-10x:Same as above but with 10 repetitions per sequence, creating much longer inputs\.
4. 4\.MahaNER\-random\-repeat\-4x:Four randomly chosen training sentences are combined into one sequence\. This exposes the model to diverse contexts within a single training step\.
5. 5\.MahaNER\-random\-repeat\-10x:Ten randomly chosen sentences are combined\. This provides maximal contextual diversity within each training sequence\.

Figure[2](https://arxiv.org/html/2607.23344#S4.F2)illustrates the difference between the two augmentation strategies\.

Self\-Concat \(SC\)Sentence ASentence ASentence ASentence ASame sentence×\\times4Random\-Concat \(RC\)Sentence ASentence BSentence CSentence D4 different sentencesFigure 2:Illustration of the two augmentation strategies\. Self\-concatenation \(SC\) repeats the same sentence multiple times\. Random concatenation \(RC\) combines different sentences into one longer sequence\.All models were fine\-tuned using the same hyperparameter settings for fair comparison\. A linear classification head was placed on top of the MahaBERT encoder, and training was performed using cross\-entropy loss over non\-IOB labels\.

### 4\.2Model Architecture

Figure[3](https://arxiv.org/html/2607.23344#S4.F3)shows the architecture of the fine\-tuned MahaBERT NER model used in this study\.

Input: Marathi Sentence \(tokens\)Subword Tokenisation \(MahaBERT vocab\)Token \+ Position EmbeddingsMahaBERT\-v2 Encoder \(12 layers, 768\-dim\)Contextual Token RepresentationsLinear Classification Head \(768→\\toNon\-IOB labels\)Output: Non\-IOB NER Tag per TokenFigure 3:Architecture of the fine\-tuned MahaBERT\-v2 NER model\. The self\-attention mechanism of the transformer encoderVaswaniet al\.\([2017](https://arxiv.org/html/2607.23344#bib.bib5)\)produces contextual representations for each token, which are then passed to a linear classification head that predicts non\-IOB NER labels\.
### 4\.3Large Language Model Evaluation

Four LLMs were evaluated on the same test sentences in a zero\-shot setting:

- •Gemini:Google’s general\-purpose multimodal LLM\.
- •LLaMA\-3\.3\-70B\-VersatileDubeyet al\.\([2024](https://arxiv.org/html/2607.23344#bib.bib7)\): Meta’s open\-weight large language model with 70 billion parameters\.
- •Gemma\-2\-9B\-ITGemma Teamet al\.\([2024](https://arxiv.org/html/2607.23344#bib.bib8)\): A smaller instruction\-tuned model from Google\.
- •Gemma\-2\-27B\-ITGemma Teamet al\.\([2024](https://arxiv.org/html/2607.23344#bib.bib8)\): A larger instruction\-tuned Gemma model\.

All LLMs were given the same prompt template for each test sentence, inspired by zero\-shot prompting strategies fromBrownet al\.\([2020](https://arxiv.org/html/2607.23344#bib.bib9)\)\. The prompt instructed the model to label each token in the sentence with its NER tag in non\-IOB format\. No examples were provided in the prompt \(zero\-shot\)\. Token alignment was enforced to ensure that model outputs could be compared directly against the gold labels using the same evaluation metrics\.

## 5Evaluation Protocol

### 5\.1Metrics

All models were evaluated using the following standard NER metrics computed at the token level:

- •Precision:Of all tokens that the model labelled as entities, what fraction were actually entities\.
- •Recall:Of all tokens that were actually entities, what fraction did the model correctly identify\.
- •F1\-score:The harmonic mean of precision and recall, giving a single balanced measure of performance\.

### 5\.2Fairness of Comparison

To ensure a fair comparison between fine\-tuned models and LLMs, the following steps were taken:

- •All models were tested on exactly the same 500 Marathi sentences\.
- •All LLMs received the same prompt template with the same formatting instructions\.
- •Token alignment was enforced for LLM outputs to prevent mismatches between model output tokens and gold\-standard tokens\.
- •No fine\-tuning, few\-shot examples, or additional context was provided to the LLMs\.

## 6Results

Tables[3](https://arxiv.org/html/2607.23344#S6.T3)–[5](https://arxiv.org/html/2607.23344#S6.T5)present the token\-level Precision, Recall, and F1\-score for all evaluated models across the three benchmark datasets\.

### 6\.1Results on Normal Dataset

Table 3:Performance on the normal \(standard\) test dataset\.On the normal test set, the MahaNER Base Model achieves the highest F1\-score of 0\.91, closely followed by the MahaNER\-normal\-repeat\-4x, MahaNER\-random\-repeat\-4x, and MahaNER\-random\-repeat\-10x variants at 0\.90\. The existing MahaNER baseline achieves 0\.8843\. All fine\-tuned models significantly outperform all LLMs\. Gemini achieves the best LLM F1 at 0\.69, while Gemma\-2\-27B performs the worst at 0\.57\.

### 6\.2Results on Normal Repeat 4x Dataset

Table 4:Performance on the Normal Repeat 4 test dataset\.On the repeated test set, MahaNER\-normal\-repeat\-4x and MahaNER\-normal\-repeat\-10x both achieve F1 of 0\.91\. This suggests that training with repeated sequences helps models handle longer, repetitive inputs\. The existing MahaNER baseline drops slightly to 0\.8668 on this variant, while Gemini’s F1 improves marginally to 0\.70\.

### 6\.3Results on Random Repeat 4x Dataset

Table 5:Performance on the Random Repeat 4 test dataset\.On the random repeat test set, MahaNER\-random\-repeat\-4x achieves the best F1 of 0\.89\. This is expected, since training with random concatenation better prepares the model for sequences containing diverse entity contexts\. LLM performance drops notably on this dataset, with LLaMA and Gemma models falling to F1 scores between 0\.35 and 0\.49\.

### 6\.4Summary of Results

Table[6](https://arxiv.org/html/2607.23344#S6.T6)provides a summary comparison of the best\-performing fine\-tuned model versus the best LLM across all three test sets\.

Table 6:Best fine\-tuned model vs best LLM \(Gemini\) across all test sets\.Across all three test sets, fine\-tuned MahaBERT models outperform the best LLM \(Gemini\) by more than 0\.20 F1 points\. This gap is consistent and large, confirming that task\-specific fine\-tuning on Marathi data remains far superior to zero\-shot LLM inference for Marathi NER\.

## 7Discussion

### 7\.1Fine\-tuned Models vs LLMs

The experimental results consistently show that fine\-tuned MahaBERT models outperform all evaluated LLMs across every test set\. The best fine\-tuned model achieves an F1\-score of 0\.91, while the best LLM \(Gemini\) reaches only 0\.69 to 0\.70 depending on the test set\. This is a gap of more than 20 F1 points, which is very large in NER evaluation terms\.

The main reason for this gap is that fine\-tuned models have been trained directly on Marathi NER data\. They have learned the specific entity patterns, label distributions, and linguistic structures present in Marathi text\. LLMs, on the other hand, are general\-purpose models that have not been explicitly trained to perform token\-level NER in Marathi\. Even though LLMs are much larger and trained on more text overallBrownet al\.\([2020](https://arxiv.org/html/2607.23344#bib.bib9)\), they lack the task\-specific and language\-specific supervision needed for high\-quality Marathi NER\. This is consistent with prior findings that cross\-lingual models underperform language\-specific ones on Marathi tasksConneauet al\.\([2020](https://arxiv.org/html/2607.23344#bib.bib6)\)\.

### 7\.2Effect of Data Augmentation

The augmentation experiments reveal a consistent pattern: models trained with augmented data generalise better to the corresponding augmented test sets\. MahaNER\-normal\-repeat\-4x and MahaNER\-normal\-repeat\-10x models excel on the Normal Repeat 4x test set, while MahaNER\-random\-repeat\-4x achieves the best score on the Random Repeat 4x test set\. This suggests that pretraining\-style data diversity strategiesLiuet al\.\([2019](https://arxiv.org/html/2607.23344#bib.bib11)\); Raffelet al\.\([2020](https://arxiv.org/html/2607.23344#bib.bib14)\)are also beneficial when applied to fine\-tuning data for low\-resource NER tasks\.

### 7\.3LLM Behaviour

Among the LLMs, Gemini consistently performs the best, with recall values above 0\.78 on all test sets\. However, Gemini’s precision is low \(around 0\.61\), meaning it often labels tokens as entities when they are not\. This high\-recall, low\-precision pattern suggests that Gemini tends to over\-predict entities rather than miss them\. LLaMADubeyet al\.\([2024](https://arxiv.org/html/2607.23344#bib.bib7)\)and GemmaGemma Teamet al\.\([2024](https://arxiv.org/html/2607.23344#bib.bib8)\)models show lower performance overall, especially on the random repeat test set where their F1 scores fall to as low as 0\.35\.

A key challenge for LLMs in this task is consistent token\-level output formatting\. Unlike discriminative models which always produce one label per token, generative LLMs may skip tokens, merge them, or produce labels in inconsistent formats\. Chain\-of\-thought promptingWeiet al\.\([2022](https://arxiv.org/html/2607.23344#bib.bib10)\)might help but was not explored in this zero\-shot evaluation\. These alignment issues directly hurt precision and recall scores\.

## Limitations

While this study provides a thorough comparison between fine\-tuned models and LLMs, there are several limitations that should be noted:

- •Zero\-shot LLM evaluation only:All LLMs were evaluated in a zero\-shot setting\. Providing a few labelled examples \(few\-shot prompting\)Brownet al\.\([2020](https://arxiv.org/html/2607.23344#bib.bib9)\)or fine\-tuning LLMs directly on Marathi NER data might yield better LLM results\. This comparison is left for future work\.
- •Single domain:All data comes from Marathi news textLitakeet al\.\([2022](https://arxiv.org/html/2607.23344#bib.bib3)\)\. Models trained and tested on this domain may not generalise well to other domains such as social mediaChaudhariet al\.\([2023](https://arxiv.org/html/2607.23344#bib.bib4)\), legal text, or conversational speech\.
- •Limited augmentation strategies:The augmentation strategies explored in this paper are limited to sentence repetition and random sentence combination\. Other augmentation methods such as back\-translation, synonym replacement, or entity substitution were not explored\.
- •Fixed prompt template for LLMs:All LLMs used the same prompt\. Different prompt designs or chain\-of\-thought strategiesWeiet al\.\([2022](https://arxiv.org/html/2607.23344#bib.bib10)\)might yield different results\. Prompt sensitivity analysis was not conducted\.
- •Non\-IOB scheme:The experiments use a non\-IOB tagging scheme\. Results may differ under other schemes such as IOB2 or BIOES, which are commonly used in benchmarks like CoNLL\-2003Sang and Meulder \([2003](https://arxiv.org/html/2607.23344#bib.bib15)\)\.

## 8Conclusion

This paper presented a comparative evaluation of fine\-tuned MahaBERT\-based Marathi NER models against modern general\-purpose Large Language Models, including Gemini, LLaMA\-3\.3\-70BDubeyet al\.\([2024](https://arxiv.org/html/2607.23344#bib.bib7)\), and Gemma variantsGemma Teamet al\.\([2024](https://arxiv.org/html/2607.23344#bib.bib8)\)\. We evaluated all systems on three MahaNER\-derived test datasetsLitakeet al\.\([2022](https://arxiv.org/html/2607.23344#bib.bib3)\): a standard benchmark, a repeated\-sentence dataset, and a random\-combination dataset\.

The main findings of this study are:

- •Fine\-tuned MahaBERT modelsJoshi \([2022a](https://arxiv.org/html/2607.23344#bib.bib2)\)consistently and significantly outperform all evaluated LLMs on all three test sets\. The best fine\-tuned model achieves an F1\-score of 0\.91, compared to 0\.69–0\.70 for the best LLM\.
- •Data augmentation through random concatenation \(MahaNER\-random\-repeat\-4x/10x\) improves model robustness on diverse test inputs\.
- •Self\-concatenation augmentation \(MahaNER\-normal\-repeat\-4x/10x\) helps models generalise better to repeated\-sentence test inputs\.
- •LLMs show high recall but low precision for Marathi NER, suggesting they tend to over\-predict entity spans\.
- •Language\-specific pretraining and supervised fine\-tuning remain the most effective approach for low\-resource Marathi NER, consistent with findings from multilingual NER researchConneauet al\.\([2020](https://arxiv.org/html/2607.23344#bib.bib6)\); Liet al\.\([2022](https://arxiv.org/html/2607.23344#bib.bib12)\)\.

Future work may explore few\-shot or fine\-tuned LLM approaches for Marathi NER, additional augmentation strategies, expansion to other Marathi domains such as social media and legal textChaudhariet al\.\([2023](https://arxiv.org/html/2607.23344#bib.bib4)\), and joint entity and relation extraction systems\.

## Acknowledgements

This work was carried out under the mentorship of L3Cube, Pune\. We would like to express our gratitude towards our mentor for his continuous support and encouragement\. This work is a part of the L3Cube\-MahaNLP projectJoshi \([2022b](https://arxiv.org/html/2607.23344#bib.bib16)\)\.

## References

- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal,et al\.\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems 33 \(NeurIPS 2020\),Cited by:[§2\.3](https://arxiv.org/html/2607.23344#S2.SS3.p1.1),[§4\.3](https://arxiv.org/html/2607.23344#S4.SS3.p2.1),[§7\.1](https://arxiv.org/html/2607.23344#S7.SS1.p2.1),[1st item](https://arxiv.org/html/2607.23344#Sx1.I1.i1.p1.1)\.
- L3Cube\-mahasocialner: a social media based marathi named entity recognition dataset and bert models\.InProceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation,pp\. 93–100\.Cited by:[§2\.2](https://arxiv.org/html/2607.23344#S2.SS2.p2.1),[§8](https://arxiv.org/html/2607.23344#S8.p3.1),[2nd item](https://arxiv.org/html/2607.23344#Sx1.I1.i2.p1.1)\.
- A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov \(2020\)Unsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 8440–8451\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747)Cited by:[§2\.2](https://arxiv.org/html/2607.23344#S2.SS2.p2.1),[§7\.1](https://arxiv.org/html/2607.23344#S7.SS1.p2.1),[5th item](https://arxiv.org/html/2607.23344#S8.I1.i5.p1.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long and Short Papers\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§2\.1](https://arxiv.org/html/2607.23344#S2.SS1.p3.1),[§4\.1](https://arxiv.org/html/2607.23344#S4.SS1.p1.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle,et al\.\(2024\)The LLaMA 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2607.23344#S1.p5.1),[§2\.3](https://arxiv.org/html/2607.23344#S2.SS3.p1.1),[2nd item](https://arxiv.org/html/2607.23344#S4.I2.i2.p1.1),[§7\.3](https://arxiv.org/html/2607.23344#S7.SS3.p1.1),[§8](https://arxiv.org/html/2607.23344#S8.p1.1)\.
- Gemma Team, T\. Mesnard, C\. Hardin, R\. Dadashi, S\. Bhupatiraju, S\. Pathak, L\. Sifre,et al\.\(2024\)Gemma: open models based on Gemini research and technology\.arXiv preprint arXiv:2403\.08295\.Cited by:[§1](https://arxiv.org/html/2607.23344#S1.p5.1),[§2\.3](https://arxiv.org/html/2607.23344#S2.SS3.p1.1),[3rd item](https://arxiv.org/html/2607.23344#S4.I2.i3.p1.1),[4th item](https://arxiv.org/html/2607.23344#S4.I2.i4.p1.1),[§7\.3](https://arxiv.org/html/2607.23344#S7.SS3.p1.1),[§8](https://arxiv.org/html/2607.23344#S8.p1.1)\.
- R\. B\. Joshi \(2022a\)L3cube\-mahacorpus and mahabert: marathi monolingual corpus, marathi bert language models, and resources\.InProceedings of the WILDRE\-6 workshop within the 13th language resources and evaluation conference,pp\. 97–101\.Cited by:[§1](https://arxiv.org/html/2607.23344#S1.p4.2),[§2\.2](https://arxiv.org/html/2607.23344#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2607.23344#S4.SS1.p1.1),[1st item](https://arxiv.org/html/2607.23344#S8.I1.i1.p1.1)\.
- R\. Joshi \(2022b\)L3cube\-mahanlp: marathi natural language processing datasets, models, and library\.arXiv preprint arXiv:2205\.14728\.Cited by:[§1](https://arxiv.org/html/2607.23344#S1.p2.1),[Acknowledgements](https://arxiv.org/html/2607.23344#Sx2.p1.1)\.
- G\. Lample, M\. Ballesteros, S\. Subramanian, K\. Kawakami, and C\. Dyer \(2016\)Neural architectures for named entity recognition\.InProceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,San Diego, California,pp\. 260–270\.External Links:[Document](https://dx.doi.org/10.18653/v1/N16-1030)Cited by:[§2\.1](https://arxiv.org/html/2607.23344#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2607.23344#S2.SS1.p2.1)\.
- J\. Li, A\. Sun, J\. Han, and C\. Li \(2022\)A survey on deep learning for named entity recognition\.IEEE Transactions on Knowledge and Data Engineering34\(1\),pp\. 50–70\.External Links:[Document](https://dx.doi.org/10.1109/TKDE.2020.2981314)Cited by:[§1](https://arxiv.org/html/2607.23344#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.23344#S2.SS1.p1.1),[5th item](https://arxiv.org/html/2607.23344#S8.I1.i5.p1.1)\.
- O\. Litake, M\. R\. Sabane, P\. S\. Patil, A\. A\. Ranade, and R\. B\. Joshi \(2022\)L3cube\-mahaner: a marathi named entity recognition dataset and bert models\.InProceedings of the WILDRE\-6 Workshop within the 13th Language Resources and Evaluation Conference,pp\. 29–34\.Cited by:[§1](https://arxiv.org/html/2607.23344#S1.p4.2),[§2\.2](https://arxiv.org/html/2607.23344#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2607.23344#S3.SS1.p1.1),[§8](https://arxiv.org/html/2607.23344#S8.p1.1),[2nd item](https://arxiv.org/html/2607.23344#Sx1.I1.i2.p1.1)\.
- Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov \(2019\)RoBERTa: a robustly optimized BERT pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[§2\.1](https://arxiv.org/html/2607.23344#S2.SS1.p3.1),[§7\.2](https://arxiv.org/html/2607.23344#S7.SS2.p1.1)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of Machine Learning Research21\(140\),pp\. 1–67\.External Links:[Link](http://jmlr.org/papers/v21/20-074.html)Cited by:[§2\.1](https://arxiv.org/html/2607.23344#S2.SS1.p3.1),[§7\.2](https://arxiv.org/html/2607.23344#S7.SS2.p1.1)\.
- E\. F\. T\. K\. Sang and F\. D\. Meulder \(2003\)Introduction to the CoNLL\-2003 shared task: language\-independent named entity recognition\.InProceedings of the Seventh Conference on Natural Language Learning at HLT\-NAACL 2003,Edmonton, Canada,pp\. 142–147\.Cited by:[§1](https://arxiv.org/html/2607.23344#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.23344#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.23344#S3.SS1.p1.1),[5th item](https://arxiv.org/html/2607.23344#Sx1.I1.i5.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems 30 \(NeurIPS 2017\),pp\. 5998–6008\.Cited by:[§2\.1](https://arxiv.org/html/2607.23344#S2.SS1.p3.1),[Figure 3](https://arxiv.org/html/2607.23344#S4.F3)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems 35 \(NeurIPS 2022\),pp\. 24824–24837\.Cited by:[§2\.3](https://arxiv.org/html/2607.23344#S2.SS3.p1.1),[§7\.3](https://arxiv.org/html/2607.23344#S7.SS3.p2.1),[4th item](https://arxiv.org/html/2607.23344#Sx1.I1.i4.p1.1)\.