Nepali Legal Expertise through Generative and Extractive Pre-trained Transformers (NepLEGiT)

# Nepali Legal Expertise through Generative and Extractive Pre-trained Transformers (NepLEGiT)
Source: [https://arxiv.org/html/2609.16010](https://arxiv.org/html/2609.16010)
Ranjit Raut Department of Artificial Intelligence Kathmandu University Dhulikhel, Nepal rautranjit916@gmail\.com &Tishya Dhakal Department of Artificial Intelligence Kathmandu University Dhulikhel, Nepal tishys57@gmail\.com &Aaryan Shakya Department of Artificial Intelligence Kathmandu University Dhulikhel, Nepal nayranewar33@gmail\.com &Bhabuk Thapa Department of Artificial Intelligence Kathmandu University Dhulikhel, Nepal vhabukthapa@gmail\.com &Prasiddha Koirala Department of Artificial Intelligence Kathmandu University Dhulikhel, Nepal prasiddhaf23@gmail\.com &Bal Krishna Bal Department of Computer Science and Engineering Kathmandu University Dhulikhel, Nepal bal@ku\.edu\.np

###### Abstract

The complexity of legal language and limited accessibility to legal information pose significant challenges to justice delivery in Nepal\. Traditional legal services remain inaccessible to many citizens due to language barriers, information fragmentation, and a critical shortage of legal expertise, particularly in rural areas\. We presentNepLEGiT\(NepaliLegalExpertise throughGenerative andExtractive Pre\-trainedTransformers\), a specialized small language model \(SLM\) designed to democratize legal knowledge and enhance legal\-service delivery in Nepal\. We pre\-train a decoder\-based GPT\-2 SLM from scratch on a curated corpus of∼\\sim4 million tokens of Nepali legal text, covering constitutional law, civil and criminal codes, and administrative regulations\. The model comprises∼\\sim30 million parameters in a 6\-layer, 6\-head, 384\-dimensional transformer trained with warmup cosine\-decay scheduling, gradient accumulation, and mixed\-precision arithmetic\. On a held\-out validation split, NepLEGiT attains a cross\-entropy loss of 0\.5684, a perplexity of 1\.8, and a next\-token prediction accuracy of 82\.9%\. We further evaluate continual masked\-language\-model pre\-training of mBERT and MuRIL on the same corpus; mBERT achieves a perplexity of 2\.35 \(eval loss 0\.8565\), outperforming MuRIL \(perplexity 6\.07, eval loss 1\.8026\), providing a strong encoder baseline complementary to NepLEGiT’s generative orientation\.

## 1Introduction

The digitization of legal knowledge has catalyzed a wave of legal AI systems tailored to document analysis, question answering, and compliance assistance\. Yet the overwhelming majority of these systems target English\-language, common\-law jurisdictions, leaving the legal systems of the Global South largely unaddressed\. Nepal exemplifies this gap: its legal framework blends elements of civil law, common law, and indigenous customary practices, written primarily in formal Nepali suffused with Sanskrit\-derived terminology that places it beyond the effective reach of general\-purpose multilingual models\.

The judiciary of Nepal handles an ever\-growing caseload under conditions of sparse digitization and acute shortages of qualified legal professionals in rural districts\. Citizens who cannot afford lawyers often navigate the legal system without assistance, compounding existing inequalities\. Meanwhile, the country’s Digital Nepal Framework articulates a mandate for e\-governance and digitally empowered citizenship, a mandate that AI\-powered legal tools could help fulfill\.

The challenges of Nepal’s legal accessibility include: \(i\) Language barriers: legal documents use formal Nepali with Sanskrit\-derived terminology that hinders comprehension for average citizens; \(ii\) Limited legal expertise: legal information is scattered across gazettes, court decisions, and orders, with severe shortages of qualified professionals in remote districts; \(iii\) Technological gap: existing legal AI targets Western \(common/civil law\) systems and English, ignoring Nepal’s mixed jurisprudence; and \(iv\) Compliance complexity: rapidly changing regulations cause inadvertent violations among businesses and citizens\. We address this need with a 10M\-token dataset sourced from the Nepal Law Commission, with a 4M\-token clean training split, a GPT\-2 decoder\-based language model pre\-trained from scratch on a curated Nepali legal corpus, and continual pre\-training masked\-language\-model mBERT and MuRIL\.

## 2Related Work

Domain\-specific pre\-training on legal corpora has consistently outperformed general\-purpose models on legal NLP tasks\. LegalBERTZheng et al\. \([2021](https://arxiv.org/html/2609.16010#bib.bib15)\)demonstrated this for English by continuing BERT pre\-training on contracts, court opinions, and statutory texts, yielding improvements on document classification and named\-entity recognition\. LexGLUEChalkidis et al\. \([2022](https://arxiv.org/html/2609.16010#bib.bib2)\)consolidated evaluation for English legal NLP across classification, NER, QA, and entailment tasks, establishing that in\-domain pre\-training is the most robust single intervention for legal model performance\. Harvey AI and Legal Robot extend this to GPT\-4\-powered generative settings, though both remain restricted to common\-law, English\-language contexts\.

mBERT and XLM\-RConneau et al\. \([2020](https://arxiv.org/html/2609.16010#bib.bib3)\)provide broad multilingual coverage but underperform in highly specialized domains such as law, where vocabulary and syntactic conventions diverge sharply from general\-domain training data\. MuRILKhanuja et al\. \([2021](https://arxiv.org/html/2609.16010#bib.bib6)\)demonstrated that language\-specific fine\-tuning substantially benefits Indic languages including Nepali, supporting our choice to pre\-train on a Nepali\-only corpus\. In our experiments, continual masked\-language\-model pre\-training of mBERT on the Nepali legal corpus yields a perplexity of 2\.35, outperforming MuRIL \(perplexity 6\.07\) on this domain — an important empirical finding suggesting that broad multilingual coverage can be more amenable to legal domain adaptation than Indic\-focused pre\-training when the target language is Nepali\. No prior work, however, has addressed Nepali legal NLP at the pre\-training stage\.

JEC\-QAZhong et al\. \([2020](https://arxiv.org/html/2609.16010#bib.bib16)\)provides a landmark dataset for Chinese legal question answering and demonstrates that non\-Western legal systems with distinct jurisdictions, unique citation conventions, and non\-Latin scripts require bespoke solutions\. COLIEERabelo et al\. \([2019](https://arxiv.org/html/2609.16010#bib.bib10)\)advances case\-retrieval for common law but does not generalize beyond English\. NepLEGiT is positioned analogously to JEC\-QA but for the Nepali legal and linguistic context\.

Eldan and Li \([2023](https://arxiv.org/html/2609.16010#bib.bib5)\)showed that small models trained on carefully curated domain\-specific data can match or exceed much larger general\-purpose models on targeted tasks\. This motivates our choice to pre\-train a∼\\sim30M\-parameter model rather than fine\-tune a multi\-billion\-parameter LLM: parameter efficiency, fast inference, and deployability under resource constraints are essential for legal AI in a developing\-country context\.

The transformerVaswani et al\. \([2017](https://arxiv.org/html/2609.16010#bib.bib14)\)introduced self\-attention as the foundation for all contemporary language models\. BERTDevlin et al\. \([2019](https://arxiv.org/html/2609.16010#bib.bib4)\)and RoBERTaLiu et al\. \([2019](https://arxiv.org/html/2609.16010#bib.bib8)\)popularized bidirectional encoder pre\-training; GPT and its successorsRadford et al\. \([2018](https://arxiv.org/html/2609.16010#bib.bib11),[2019](https://arxiv.org/html/2609.16010#bib.bib12)\); Brown et al\. \([2020](https://arxiv.org/html/2609.16010#bib.bib1)\)established autoregressive decoder pre\-training as the dominant paradigm for text generation\. BARTLewis et al\. \([2020](https://arxiv.org/html/2609.16010#bib.bib7)\)and T5Raffel et al\. \([2020](https://arxiv.org/html/2609.16010#bib.bib13)\)combined both into encoder–decoder architectures suited to seq2seq tasks\. Our architecture evaluation \(Section[3\.2](https://arxiv.org/html/2609.16010#S3.SS2)\) leads us to select GPT\-2 for NepLEGiT’s generative orientation\.

## 3Methodology

### 3\.1Dataset

We made a Nepali legal corpus from the Nepal Law Commission: Constitution, civil and criminal codes, and all major statutes\.

Table 1:Nepali Legal Corpus Statistics
### 3\.2Models

Bidirectional encoding excels at understanding tasks: classification, NER, extractive QA, but the absence of a causal decoder renders it ill\-suited for open\-ended legal text generation, the primary use case of NepLEGiT\. We select mBERT and MuRIL for NepLEGiT\. Sequence\-to\-sequence architectures offer versatility across comprehension and generation and are well\-suited to summarization and generative QA\. Their dual\-stack design doubles computational cost and requires substantially larger datasets to converge effectively\. Autoregressive decoders produce coherent, context\-consistent outputs over extended sequences\. The causal language\-modelling objective is straightforward to optimize, scales predictably, and directly serves NepLEGiT’s primary use cases: legal document drafting, explanation generation, and conversational legal guidance\. We select decoder\-based GPT\-2 for NepLEGiT\. We evaluated three transformer families against the requirements of Nepali legal text generation\.

![Refer to caption](https://arxiv.org/html/2609.16010v1/system-architecture.png)Figure 1:System Architecture\.We adopt the GPT\-2 BPE tokenizer \(Tiktoken library, vocabulary size 50,257\) for its robustness to Unicode characters in NepLEGiT’s GPT\-2 architecture\. After NFC normalization, it handles Nepali subword segmentation acceptably; average legal document length is∼\\sim2,500 tokens\. The preprocessed corpus was serialized to binary NumPymemmapformat, enabling efficient random\-access I/O without loading the full dataset into RAM\. In the context of mBERT and MuRIL, we used their own dedicated tokenizer config and vocabulary hosted on Hugging Face, which we loaded using the standard AutoTokenizer classes\.

![Refer to caption](https://arxiv.org/html/2609.16010v1/model-architecture.png)Figure 2:GPT\-2 Model Architecture\.NepLEGiT implements a standard causal transformer decoder followingVaswani et al\. \([2017](https://arxiv.org/html/2609.16010#bib.bib14)\)with the modifications ofRadford et al\. \([2019](https://arxiv.org/html/2609.16010#bib.bib12)\)\. Table[2](https://arxiv.org/html/2609.16010#S3.T2)summarizes the configuration\.

Table 2:NepLEGiT GPT\-2 Model ConfigurationLayer normalization is applied*before*the attention and MLP sub\-layers \(rather than after\), stabilizing gradient flow in deep networksBrown et al\. \([2020](https://arxiv.org/html/2609.16010#bib.bib1)\)\. Feed\-forward sub\-layers use GELU with a4×4\\timeshidden\-dimension expansion \(4×384=1,5364\\times 384=1\{,\}536\)\. The output projection matrix is tied to the input token embedding, reducing parameters and enforcing representational consistency between input and output spaces\. Scaled dot\-product self\-attention is masked to prevent attention to future positions\. Flash attention is enabled where hardware supports it\. Summed with token embeddings at the input layer\. Linear layers use𝒩​\(0,0\.02\)\\mathcal\{N\}\(0,0\.02\); residual projection layers use𝒩​\(0,0\.02/2​nlayer\)\\mathcal\{N\}\(0,\\,0\.02/\\\!\\sqrt\{2\\,n\_\{\\text\{layer\}\}\}\)followingRadford et al\. \([2019](https://arxiv.org/html/2609.16010#bib.bib12)\)\.

### 3\.3Training

Table 3:Training HyperparametersWe use AdamW with decoupled weight decayLoshchilov and Hutter \([2018](https://arxiv.org/html/2609.16010#bib.bib9)\)\. The higherβ2=0\.95\\beta\_\{2\}=0\.95\(versus the conventional 0\.999\) produces smoother second\-moment estimates that benefit sparse, domain\-specific vocabulary updates\. We implement a two\-phase schedule via PyTorch’sSequentialLR\.

Phase 1 – Linear warmup \(steps 0–1,000\)

ηt=ηmax⋅ttwarm\\eta\_\{t\}=\\eta\_\{\\max\}\\cdot\\frac\{t\}\{t\_\{\\mathrm\{warm\}\}\}Gradual ramp\-up prevents large gradient updates from random initialization from destabilizing early training\.

Phase 2 – Cosine annealing decay \(steps 1,000–100,000\)

ηt=ηmin\+12​\(ηmax−ηmin\)​\(1\+cos⁡\(π​t−twarmT−twarm\)\)\\eta\_\{t\}=\\eta\_\{\\min\}\+\\tfrac\{1\}\{2\}\(\\eta\_\{\\max\}\-\\eta\_\{\\min\}\)\\\!\\left\(1\+\\cos\\\!\\left\(\\pi\\,\\frac\{t\-t\_\{\\mathrm\{warm\}\}\}\{T\-t\_\{\\mathrm\{warm\}\}\}\\right\)\\right\)The cosine envelope provides a smooth, monotone decay, avoiding abrupt rate changes while continuing slow refinement atηmin=5×10−5\\eta\_\{\\min\}=5\\times 10^\{\-5\}\. To simulate an effective batch size of 1,024 on hardware limited to micro\-batches of 32, gradients are accumulated over 32 steps before each optimizer update\. Automatic mixed precision \(AMP\) with bfloat16 \(float16 fallback\) halves memory consumption and increases throughput\. AGradScalerprevents underflow in float16 gradient computations\. Memory\-mapped binary files \(NumPyuint16memmap\) allow random\-access sampling across the full corpus without loading it into RAM\. Each step samples a random batch of context windows and constructs next\-token prediction targets by shifting by one position\. Every 500 training iterations, the model is placed in evaluation mode, and cross\-entropy loss is averaged over 500 random validation batches\. The checkpoint achieving the lowest validation loss is retained for final evaluation\. Training was conducted on cloud instances \(Tesla T4, P100\) via Kaggle, Colab, and Lightning AI\. The full 100,000\-iteration run required approximately∼\\sim12 hours of wall\-clock time and a peak VRAM footprint of∼\\sim12 GB\. The mBERT and MuRIL continual pre\-training experiments were conducted on a Tesla P100\-PCIE\-16 GB \(Kaggle\), requiring approximately 2–3 hours each for 3 epochs over∼\\sim27K training examples\.

## 4Results

Three distinct phases characterize training\. Rapid loss reduction from≈\\approx7\.5 to≈\\approx2\.0, driven by warmup initialization and steep gradient descent from 0–20,000 iterations\. Sustained steady improvement as cosine decay refines learned representations from 20,000–50,000 iterations, and from 50,000–100,000 iterations, loss converges and stabilizes, resulting in a final training loss of≈\\approx0\.57, and a final validation loss of 0\.5684\.

![Refer to caption](https://arxiv.org/html/2609.16010v1/cross-entropy-loss.png)Figure 3:Training \(green\) and validation \(red\) cross\-entropy loss vs\. evaluation step \(one step==500 iterations\)\. Three phases: rapid descent \(0–20k iterations\), steady improvement \(20–50k\), and convergence near 0\.57 \(50–100k\)\. The tight coupling of the two curves indicates strong generalization with negligible overfitting\.PerplexityPPL=exp⁡\(ℒ\)\\mathrm\{PPL\}=\\exp\(\\mathcal\{L\}\)provides an interpretable measure of per\-token uncertainty \(Figure[4](https://arxiv.org/html/2609.16010#S4.F4)\)\. Starting from≈\\approx1,800 at initialization, it falls sharply within the first 10,000 iterations before stabilizing at 1\.8, meaning the model places the correct next token among its top\-2 predictions on average\.

![Refer to caption](https://arxiv.org/html/2609.16010v1/perplexity-score.jpg)Figure 4:Validation perplexity \(lower is better\) over training\. The reduction from∼\\sim1,800 to 1\.8 reflects deep adaptation to Nepali legal language patterns\.Token prediction accuracy follows a characteristic learning curve, converging at 82\.9%, a strong result given the specialized vocabulary and complex syntax of Nepali legal text \(Figure[5](https://arxiv.org/html/2609.16010#S4.F5)\)\.

![Refer to caption](https://arxiv.org/html/2609.16010v1/next-token-accuracy.png)Figure 5:Top\-1 next\-token prediction accuracy on the validation set\. Progression:≈\\approx38% \(init\)→\\to≈\\approx50% \(2k iter\)→\\to≈\\approx75% \(10k iter\)→\\to82\.9% \(100k iter\)\.![Refer to caption](https://arxiv.org/html/2609.16010v1/learning-rate-schedule.jpg)Figure 6:Two\-phase learning rate schedule \(log scale\): linear warmup over steps 0–1,000, followed by cosine annealing decay over steps 1,000–100,000\.Table[4](https://arxiv.org/html/2609.16010#S4.T4)summarizes the final metrics on the held\-out validation split\.

Table 4:Final Model Performance on Held\-Out Validation SetTable[5](https://arxiv.org/html/2609.16010#S4.T5)compares NepLEGiT against all evaluated baselines on Nepali legal text, including the two encoder models subjected to continual masked\-language\-model pre\-training\.

Table 5:Comparison Against Baseline Models on Nepali Legal TextNepLEGiT’s perplexity of 1\.8 represents a 22–33×\\timesimprovement over GPT\-2 Small despite using only 26% of its parameters\. Among the encoder models, mBERT \(perplexity 2\.35\) substantially outperforms MuRIL \(perplexity 6\.07\) after continual pre\-training on the same corpus\. These findings collectively confirm that targeted in\-domain pre\-training from scratch substantially outweighs scaling for specialized low\-resource language tasks, consistent with the findings ofEldan and Li \([2023](https://arxiv.org/html/2609.16010#bib.bib5)\)\.

To complement NepLEGiT’s generative decoder approach, we performed continual masked\-language\-model \(MLM\) pre\-training of two established multilingual encoder models: mBERT \(bert\-base\-multilingual\-cased\) and MuRIL \(google/muril\-base\-cased\) on the same Nepali legal corpus\. Both models were fine\-tuned for 3 epochs using the Hugging FaceTrainerAPI on a Tesla P100\-PCIE\-16 GB GPU, with the following shared configuration: batch size 2, gradient accumulation steps 16 \(effective batch 32\), learning rate1×10−51\\times 10^\{\-5\}, cosine schedule, weight decay 0\.01, max sequence length 256, and MLM probability 0\.20\.

Table 6:NepLEGiT Models on Masked Language ModelingTable[7](https://arxiv.org/html/2609.16010#S4.T7)reports the validation loss trajectory at selected checkpoints\.

Table 7:Validation Loss During Continual MLM Pre\-training \(mBERT vs\. MuRIL\)mBERT converges to a final eval loss of 0\.8565 \(perplexity 2\.35\), while MuRIL converges to 1\.8026 \(perplexity 6\.07\)\. mBERT is therefore the stronger encoder baseline on this domain, despite MuRIL’s explicit Indic\-language focus\. It is important to note that these perplexity values arise from a masked\-language\-modelling \(MLM\) objective and are not directly comparable to NepLEGiT’s causal language\-modelling \(CLM\) perplexity; the encoder models predict only 20% of randomly masked tokens, while NepLEGiT predicts every next token autoregressively\. These models serve distinct downstream roles: mBERT and MuRIL are better suited for classification, NER, and extractive QA, while NepLEGiT targets generative legal text production\.

## 5Discussion

### 5\.1Domain\-Specific Pre\-training

The baseline comparison \(Table[5](https://arxiv.org/html/2609.16010#S4.T5)\) provides clear evidence of domain specificity’s value\. GPT\-2 Small, with4×4\\timesmore parameters, achieves only 45–55% token accuracy versus NepLEGiT’s 82\.9%\. Two factors explain this gap: \(i\) GPT\-2’s tokenizer and weights encode English distributional statistics that poorly match Nepali Unicode character sequences; and \(ii\) the domain shift from general web text to formal legal Nepali spans both language and register simultaneously\. Targeted pre\-training collapses both gaps at once\.

### 5\.2mBERT vs\. MuRIL on Nepali Legal Text

A notable result of the encoder experiments is that mBERT outperforms MuRIL on the Nepali legal corpus despite the latter being explicitly pre\-trained on Indic languages including NepaliKhanuja et al\. \([2021](https://arxiv.org/html/2609.16010#bib.bib6)\)\. We attribute this to two factors\. First, MuRIL’s pre\-training emphasizes transliterated and code\-mixed Indic text, whereas Nepali legal language is formal, monolingual, and highly Sanskritized, a register that differs substantially from conversational or transliterated usage\. Second, mBERT’s larger and more diverse multilingual pre\-training may provide a more general\-purpose linguistic scaffold that adapts more readily to a new domain via continual pre\-training\. This finding has practical implications: for Nepali legal NLP tasks requiring an encoder \(e\.g\., named\-entity recognition, document classification\), mBERT is the stronger off\-the\-shelf starting point\.

### 5\.3Training Stability

The near\-identical train/validation loss curves \(Figure[3](https://arxiv.org/html/2609.16010#S4.F3)\) over 100,000 iterations indicate the model does not overfit to the training split\. Three design choices are responsible: dropout \(p=0\.1p=0\.1\) at each sub\-layer, weight decay \(λ=0\.1\\lambda=0\.1\) via AdamW, and a model capacity \(30M parameters\) that is relatively small relative to the training corpus \(4M tokens\), leaving the model slightly underfitted — which is preferable for a pre\-training stage to be followed by fine\-tuning\.

### 5\.4Perplexity Interpretation

A perplexity of 1\.8 on domain\-specific text is notably low\. GPT\-2 \(1\.5B parameters\) achieves perplexity around 18–25 on general English textRadford et al\. \([2019](https://arxiv.org/html/2609.16010#bib.bib12)\)\. The very low perplexity here reflects the high degree of formulaic repetition in legal Nepali; statutes reuse phraseological templates extensively, which a domain\-specific model captures far more efficiently than a general\-purpose one\. Caution is warranted: low next\-token perplexity does not directly translate to high downstream task performance \(QA, summarization\), which will require instruction fine\-tuning\.

### 5\.5Tokenization Limitations

The GPT\-2 BPE tokenizer segments Nepali Devanagari suboptimally, producing longer token sequences than a native Nepali tokenizer would\. This inflates effective sequence lengths and reduces contextual coverage within the 128\-token window\. We estimate that a custom Nepali BPE vocabulary \(32K–50K tokens\) trained on the legal corpus would reduce token counts by 20–35%, enabling richer context at equivalent compute\.

### 5\.6Ethical Considerations

NepLEGiT*cannot replace qualified legal professionals*\. Its outputs may contain hallucinated or factually incorrect legal information; all AI\-generated legal content must be verified by licensed practitioners\. The training corpus may encode historical biases present in Nepal’s legal documentation \(e\.g\., gender\-asymmetric precedents\)\.

## 6Conclusion

We have presented NepLEGiT from scratch, a language model pre\-trained exclusively on Nepali legal text\. Starting from a curated corpus of∼\\sim4M tokens, we trained a∼\\sim30M\-parameter GPT\-decoder SLM from scratch using mixed\-precision arithmetic, gradient accumulation, and AdamW with warmup cosine\-decay scheduling\. The model achieves a validation perplexity of 1\.8 and a next\-token accuracy of 82\.9%, outperforming a zero\-shot GPT\-2 Small baseline by a factor of 22–33×\\timesin perplexity despite using only 26% of its parameters\. As a complementary encoder\-side investigation, we performed continual pre\-training and evaluated continual masked\-language\-model pre\-training of mBERT and MuRIL on the same corpus\. mBERT achieves an MLM perplexity of 2\.35 \(eval loss 0\.8565\), outperforming MuRIL \(perplexity 6\.07\), establishing a strong encoder baseline for downstream understanding tasks\.

NepLEGiT establishes a baseline for Nepali legal NLP and provides a fully documented, replicable methodology applicable to other low\-resource, non\-Western legal systems\. The pre\-trained model is positioned as the foundation for a broader legal AI ecosystem in Nepal: continual pre\-training on multilingual backbones, supervised instruction fine\-tuning, RAG\-augmented long\-document reasoning, and a publicly accessible legal information service for Nepali citizens, legal professionals, and institutions\. By demonstrating that focused domain pre\-training at modest scale decisively outperforms general\-purpose models for this specialized task, NepLEGiT advances the case that low\-resource legal AI is tractable and consequential for equitable access to justice\. We prefer a custom Nepal legal BPE tokenizer, continual pre\-training on multilingual decoder\-based models, supervised fine\-tuning on domain\-specific downstream tasks, and RAG\-based informal retrieval systems as future work\.

## References

- Brown et al\. \(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, et al\.Language models are few\-shot learners\.*Advances in Neural Information Processing Systems*, 33:1877–1901, 2020\.
- Chalkidis et al\. \(2022\)I\. Chalkidis, A\. Jana, D\. Hartung, M\. Bommarito, I\. Androutsopoulos, D\. M\. Katz, and N\. Aletras\.LexGLUE: A benchmark dataset for legal language understanding in English\.In*Proceedings of the 60th Annual Meeting of the ACL*, pages 4310–4330, 2022\.
- Conneau et al\. \(2020\)A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, et al\.Unsupervised cross\-lingual representation learning at scale\.In*Proceedings of the 58th Annual Meeting of the ACL*, pages 8440–8451, 2020\.
- Devlin et al\. \(2019\)J\. Devlin, M\.\-W\. Chang, K\. Lee, and K\. Toutanova\.BERT: Pre\-training of deep bidirectional transformers for language understanding\.In*Proceedings of NAACL\-HLT 2019*, pages 4171–4186, 2019\.
- Eldan and Li \(2023\)R\. Eldan and Y\. Li\.TinyStories: How small can language models be and still speak coherent English?*arXiv preprint arXiv:2305\.07759*, 2023\.
- Khanuja et al\. \(2021\)S\. Khanuja, D\. Bansal, S\. Mehtani, S\. Khosla, A\. Dey, B\. Gopalan, et al\.MuRIL: Multilingual representations for Indian languages\.*arXiv preprint arXiv:2103\.10730*, 2021\.
- Lewis et al\. \(2020\)M\. Lewis, Y\. Liu, N\. Goyal, M\. Ghazvininejad, A\. Mohamed, O\. Levy, V\. Stoyanov, and L\. Zettlemoyer\.BART: Denoising sequence\-to\-sequence pre\-training for natural language generation, translation, and comprehension\.In*Proceedings of the 58th Annual Meeting of the ACL*, pages 7871–7880, 2020\.
- Liu et al\. \(2019\)Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, et al\.RoBERTa: A robustly optimized BERT pretraining approach\.*arXiv preprint arXiv:1907\.11692*, 2019\.
- Loshchilov and Hutter \(2018\)I\. Loshchilov and F\. Hutter\.Decoupled weight decay regularization\.In*International Conference on Learning Representations \(ICLR\)*, 2018\.
- Rabelo et al\. \(2019\)J\. Rabelo, M\.\-Y\. Kim, R\. Goebel, M\. Yoshioka, Y\. Kano, and K\. Satoh\.COLIEE\-2019: Machine reading comprehension task for legal texts\.In*Proceedings of the 2nd Workshop on Automated Semantic Analysis of Information in Legal Texts*, pages 61–67, 2019\.
- Radford et al\. \(2018\)A\. Radford, K\. Narasimhan, T\. Salimans, and I\. Sutskever\.Improving language understanding by generative pre\-training\.*OpenAI Technical Report*, 2018\.
- Radford et al\. \(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever\.Language models are unsupervised multitask learners\.*OpenAI Blog*, 1\(8\):9, 2019\.
- Raffel et al\. \(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, et al\.Exploring the limits of transfer learning with a unified text\-to\-text transformer\.*Journal of Machine Learning Research*, 21\(140\):1–67, 2020\.
- Vaswani et al\. \(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin\.Attention is all you need\.In*Advances in Neural Information Processing Systems*, volume 30, pages 5998–6008, 2017\.
- Zheng et al\. \(2021\)H\. Zheng, N\. Guha, B\. R\. Anderson, P\. Henderson, and D\. E\. Ho\.When does pretraining help? Assessing self\-supervised learning for law and the CaseHOLD dataset\.In*Proceedings of the 18th International Conference on Artificial Intelligence and Law*, pages 159–168, 2021\.
- Zhong et al\. \(2020\)H\. Zhong, C\. Xiao, C\. Tu, T\. Zhang, Z\. Liu, and M\. Sun\.JEC\-QA: A legal\-domain question answering dataset\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 34, pages 9701–9708, 2020\.