Polite on the Surface, Wrong in Practice: A Curated Dataset for Fixing Honorific Failures in Multilingual Bangla Generation

# Polite on the Surface, Wrong in Practice: A Curated Dataset for Fixing Honorific Failures in Multilingual Bangla Generation
Source: [https://arxiv.org/html/2605.22487](https://arxiv.org/html/2605.22487)
Md\. Asaduzzaman Shuvo,Mahedi Hasan, Md\. Tashin Parvez,Azizul Haque Noman,Md\. Shafayet Hossain Ovi,

United International University, Bangladesh Emails:\{ashuvo221104,mhasan221119\}@bscse\.uiu\.ac\.bd; \{tashinparvez2002,azizulhaquenoman,shafayethossain5463\}@gmail\.com

###### Abstract

Recent advances in Multilingual Large Language Models \(MLLMs\) have significantly enhanced cross\-lingual conversational capabilities, yet modeling culturally nuanced and context\-dependent communication remains a critical bottleneck\. Specifically, existing state\-of\-the\-art models exhibit a severe pragmatic gap when handling structural variations, regional idioms, and honorific consistencies in low\-resource contexts like Bangla\. To address this limitation, we introduce a novel, culturally aligned instruction\-tuning dataset forBangLa Application and DialoguE generation \- BLADEand benchmarking framework comprising4,1964,196meticulously curated interaction pairs\. We leverage this resource to systematically fine\-tune and evaluate leading open\-weight architectures, including DeepSeek\-8B and LLaMA\-3\.2\-3B, utilizing parameter\-efficient fine\-tuning via LoRA adapters in a 4\-bit NormalFloat \(NF4\) quantization framework\. Our empirical evaluations demonstrate that models fine\-tuned on our dataset yield substantial improvements in structural fidelity and honorific alignment, providing a rigorous benchmark for bridging pragmatic disparities in low\-resource multilingual text generation\. Code and dataset:[https://github\.com/ashuvo25/Bangla\_Application\_LLM/tree/main](https://github.com/ashuvo25/Bangla_Application_LLM/tree/main)

\[ BoldFont = TeXGyreTermesX\-Bold\.otf, ItalicFont = TeXGyreTermesX\-Italic\.otf, BoldItalicFont = TeXGyreTermesX\-BoldItalic\.otf \]

Polite on the Surface, Wrong in Practice: A Curated Dataset for Fixing Honorific Failures in Multilingual Bangla Generation

Md\. Asaduzzaman Shuvo, Mahedi Hasan,Md\. Tashin Parvez,Azizul Haque Noman,Md\. Shafayet Hossain Ovi,United International University, BangladeshEmails:\{ashuvo221104,mhasan221119\}@bscse\.uiu\.ac\.bd;\{tashinparvez2002,azizulhaquenoman,shafayethossain5463\}@gmail\.com;

## 1Introduction

Large Language Models \(LLMs\) have fundamentally transformed multilingual text generation\(Achiamet al\.,[2023](https://arxiv.org/html/2605.22487#bib.bib18); Touvronet al\.,[2023](https://arxiv.org/html/2605.22487#bib.bib19)\), yet for low\-resource languages this progress remains superficial\. Themultilingual cursemanifests not as vocabulary gaps, but as a deeperpragmatic gap, the divide between surface fluency and functional usability a structural limitation affecting any language whose institutional writing encodes social relationships grammatically\. Our findings extend beyond Bangla, the failure of general\-purpose multilingual pretraining to internalize register\-aware, format\-sensitive conventions is a structural limitation affecting any language whose institutional writing encodes social relationships grammatically\. Bangla, the fifth most spoken language worldwide with over 230 million native speakers, makes this failure concrete\. Current multilingual models produce Bangla text thatappearsfluent by surface\-level metrics while simultaneously failing every practical test of usability: misplaced document structures, inconsistent honorific registers, and culturally incongruent discourse markers that make outputs institutionally unacceptable\(Mukherjeeet al\.,[2025](https://arxiv.org/html/2605.22487#bib.bib21); Snigdha and Rahman,[2022](https://arxiv.org/html/2605.22487#bib.bib22)\)\.

Zero\-Shot \(Flawed\)BLADE\-SFT \(Correct\)মহোদয়, আমি আপনার বিদ্যালয়ের একজন ছাত্র।তুমিকি আমাকে তিন দিনের ছুটি দিতে পারবে?আপনারকাছে বিনীত আবেদন করছি।\(English: "Sir, I am a student… Canyou \[informal\]give me leave? I appeal toyou \[formal\]\."\)মহোদয়, সম্মানপূর্বক বিনীত নিবেদন এই যে, আমি আপনার বিদ্যালয়ের নবম শ্রেণীর একজন নিয়মিত ছাত্র। আগামী তিন দিনের জন্যআপনারনিকট অগ্রিম ছুটির আবেদন করছি।\(English: "Sir… I herebyappeal to you\[consistent formal\]for three days of advance leave\."\)Table 1:Comparison of generated text from a zero\-shot baseline and the DeepSeek\-8B BLADE\-SFT model\. While the zero\-shot baseline fails to maintain social context by mixing informal \(Tumi\) and formal \(Apnar\) pronouns in a professional setting, the fine\-tuned model consistently applies the correct formal honorifics required for a student\-to\-principal application\.The root cause is not vocabulary coverage it is the absence ofregister\-aware supervision\. Bangla encodes social relationships directly into its grammar through a two\-tier honorific system\. The formal second\-person pronoun আপনি \(Apni : You\) and its associated verb forms are obligatory in professional, institutional, and elder\-addressed writing\. The informal counterpart তুমি \(Tumi: You\) is reserved for peers, close friends, and juniors\. Mixing these registers within a single document is not merely stylistically awkward it signals a fundamental breakdown in social competence and renders the document unusable in any formal context\(Snigdha and Rahman,[2022](https://arxiv.org/html/2605.22487#bib.bib22)\)\. Table[1](https://arxiv.org/html/2605.22487#S1.T1)illustrates this failure concretely\. A zero\-shot baseline model, given the prompt"Write a formal leave application to your school principal", produces output that oscillates betweenApni: youandTumi: youregisters within the same paragraph an error no native speaker would make and no institution would accept\.

Beyond honorific mismatch, zero\-shot models exhibit recurring structural failures: omitting mandatory document components date, subject line, formal salutation, misplacing closings, and using discourse markers appropriate for casual speech in formal writing contexts\. These are not edge cases our empirical evaluation shows they occur systematically across all tested architectures\. Even the strongest zero\-shot baseline, Gemini\-2\.5\-Flash, achieves only 7\.88 BLEU, confirming that general\-purpose pretraining fails to internalize the format\-sensitive and register\-aware conventions required by native speakers\.

To address these limitations, we introduceBLADE \(BangLa Applications and DialoguEs\), a specialized resource serving a dual role: \(i\) an instruction\-tuning dataset of 4,196 high\-quality, expert\-annotated prompt\-response pairs spanning 2,008 unique topics, and \(ii\) a rigorous evaluative benchmark for measuring model adherence to Bangla structural and cultural norms\. BLADE was constructed through a principled three\-tier acquisition pipeline: government\-approved textbooks establish the gold standard for canonical formatting; verified web portals provide real\-world usage diversity; and author\-synthesized examples, cross\-validated by native linguistic experts, address complex honorific alignment scenarios underrepresented in static sources\.

Our contributions are as follows:

- •We introduceBLADE, a publicly available instruction\-tuning dataset of 4,196 expert\-annotated pairs spanning 2,008 unique topics, built through a three\-tier pipeline with native linguistic expert validation, covering educational, professional, and conversational Bangla generation\.
- •We benchmark the pragmatic gap in five state\-of\-the\-art multilingual LLMs under zero\-shot conditions, systematically documenting pragmatic register failures and document structure violations across all tested architectures\.
- •We demonstrate that targeted SFT onBLADEyields consistent gains across automatic metrics \(BLEU 17\.73; chrF 46\+\), human expert evaluation \(\>\>4\.6/5 on all dimensions\), and LLM\-as\-judge scoring \(8\.30/10\), establishing that cultural data specificity outweighs model scale for low\-resource pragmatic generation\.

## 2Related Work

##### Multilingual LLMs and the Low\-Resource Gap\.

Progress in LLMs has been driven by web\-scale corpora and instruction datasets such as Common Crawl,Alpaca, andDolly, with multilingual coverage expanded by projects like HPLT\(de Gibertet al\.,[2024](https://arxiv.org/html/2605.22487#bib.bib1)\)\. Despite these gains, a well\-documented 25–35% instruction\-following accuracy gap persists between high\- and low\-resource languages\(Zenget al\.,[2025](https://arxiv.org/html/2605.22487#bib.bib27)\), and LLMs consistently fail to outperform classical baselines on extremely low\-resource languages in zero\-shot settings\(Cahyawijayaet al\.,[2024](https://arxiv.org/html/2605.22487#bib.bib28)\)\. Critically, machine\-translated instruction data systematically underestimates model capability compared to natively localized equivalents\(Bawden and Yvon,[2023](https://arxiv.org/html/2605.22487#bib.bib31)\), motivating our choice to construct BLADE from human\-curated, institutionally grounded Bangla sources\.

##### Honorific Systems and Pragmatic Competence\.

A deeper multilingual failure concerns pragmatic competence: generating text that is socially and institutionally appropriate, not merely grammatical\. For languages that grammaticalize social relationships into morphology, register consistency is a necessary condition for functional usability\. Japanese LLMs fail to generalize honorific patterns across novel syntactic structures\(Sekizawa and Yanaka,[2023](https://arxiv.org/html/2605.22487#bib.bib32)\), and even frontier models including GPT\-4o underperform on high\-politeness Javanese registers underrepresented in web\-scale data\(Farhansyahet al\.,[2025](https://arxiv.org/html/2605.22487#bib.bib29)\)\. Arabic evaluation reveals the same failure: surface fluency does not imply cultural coherence\(Attiaet al\.,[2026](https://arxiv.org/html/2605.22487#bib.bib33)\)\.

Table 2:Example BLADE entry with English translation, illustrating canonical document structure\.These findings establish that Bangla’s honorific mismatch problem, documented in Table[1](https://arxiv.org/html/2605.22487#S1.T1), is a cross\-linguistically shared structural failure, not a Bangla\-specific anomaly\.

##### Bangla NLP Resources\.

Bangla NLP has progressed from encoder models such asBanglaBERT\(Bhattacharjeeet al\.,[2022](https://arxiv.org/html/2605.22487#bib.bib2)\)to native generative models includingTigerLLM\(Raihan and Zampieri,[2025](https://arxiv.org/html/2605.22487#bib.bib3)\)andTituLLMs\(Nahinet al\.,[2025](https://arxiv.org/html/2605.22487#bib.bib4)\), alongside task\-specific datasets for NER\(Paulet al\.,[2026](https://arxiv.org/html/2605.22487#bib.bib5); Mhaskeet al\.,[2023](https://arxiv.org/html/2605.22487#bib.bib8)\), captioning\(Rahmanet al\.,[2019](https://arxiv.org/html/2605.22487#bib.bib6)\), and sentiment analysis\(Islamet al\.,[2021](https://arxiv.org/html/2605.22487#bib.bib7)\)\. However, all existing resources target classification and extraction; none provides supervision for long\-form, format\-sensitive, register\-consistent generation the precise gap BLADE addresses\.

##### Data Quality over Scale\.

Our core empirical finding that a compact model fine\-tuned on BLADE outperforms zero\-shot frontier models is grounded in a broader principle:Zhouet al\.\([2023](https://arxiv.org/html/2605.22487#bib.bib26)\)show that 1,000 carefully curated examples suffice to match models trained with extensive RLHF, establishing that data quality outweighs quantity for alignment\. This has been validated in low\-resource multilingual settings, where small\-scale high\-quality SFT data outperforms larger automatically constructed corpora at the morphological and structural level\(Iyeret al\.,[2024](https://arxiv.org/html/2605.22487#bib.bib30)\)\. BLADE provides concrete evidence of this principle for register\-sensitive, format\-constrained Bangla generation\.

## 3BLADE Dataset

### 3\.1Dataset Overview

The BLADE dataset comprises 4,196 high\-quality prompt\-response pairs constructed specifically for structured Bangla generation, with an average response length exceeding 1,300 tokens\. The dataset spans 2,008 unique topics across two primary task typesapplication writinganddialogue generationcovering educational, professional, administrative, and conversational domains Table[3\.1](https://arxiv.org/html/2605.22487#S3.SS1)\.

\\rowcolorheaderblueApplicationsTopicsDialoguesTopics\\rowcolorlightblue General675General476\\rowcolorlighterblue Educational214Informal170\\rowcolorlightblue Administrative169Formal141\\rowcolorlighterblue Professional163\\rowcoloraccenttealTotal Topics2,008

Table 3:Distribution of unique topics across domains\.BLADE was designed around three core principles absent from existing Bangla corpora: \(i\)register awarenessevery example enforces consistent honorific alignment between salutations, pronouns, verbal morphology, and closings; \(ii\)structural fidelityapplication examples strictly follow canonical Bangla institutional document order \(Date→\\rightarrowAddressee→\\rightarrowSubject→\\rightarrowSalutation→\\rightarrowBody→\\rightarrowClosing→\\rightarrowSignature\); and \(iii\)topical diversitytopics represent real\-world Bangla institutional writing encountered by students, professionals, and citizens in Bangladesh\. An illustrative entry is provided in Table[2](https://arxiv.org/html/2605.22487#S2.T2)\.

### 3\.2Topic Selection

Topic inclusion followed four criteria: \(i\)real\-world prevalencein Bangladeshi educational and professional contexts, validated against source frequency across textbooks and web portals; \(ii\)register sensitivitytasks where honorific errors render output institutionally unusable; \(iii\)structural complexitys multi\-component documents beyond a single paragraph; and \(iv\)under representationin existing pretraining corpora\. Topics already well\-covered by generic web\-scale data \(e\.g\., news, Wikipedia\) were explicitly deprioritized\. The resulting distribution spans 1,221 application topics and 787 dialogue topics Table[3\.1](https://arxiv.org/html/2605.22487#S3.SS1)\.

### 3\.3Data Collection

Collection followed a three\-tier acquisition strategy, each tier governed by distinct sourcing and verification protocols\. Full annotation guidelines, annotator profiles, inter\-annotator agreement statistics, and quality control criteria are provided in Appendix[A](https://arxiv.org/html/2605.22487#A1)\.

##### Tier 1: Institutional Textbooks \(1,972 samples\)\.

Canonical application structures were extracted from nine government\-approved NCTB secondary and higher secondary textbooks, establishing a gold standard for institutional formatting\. Each entry was verified by two independent annotators for structural completeness \(all seven mandatory document components present in canonical order\) before inclusion\.

##### Tier 2: Public Web Portals \(1,382 samples\)\.

Real\-world examples were curated from 14 verified Bangla educational web portals Table[9](https://arxiv.org/html/2605.22487#A2.T9)\. Every entry underwent manual verification against structural completeness and honorific consistency criteria\. Approximately 23% of candidate web entries were discarded during this process due to register inconsistencies or malformed structures\.

##### Tier 3: Author\-Synthesized Examples \(842 samples\)\.

To address complex scenarios underrepresented in static sources particularly multi\-turn dialogues requiring sustained honorific consistency 842 examples were purpose\-built by the authors and cross\-validated by two external native linguistic experts specializing in Bangla philology\. Each Tier 3 example passed a two\-stage review: author\-annotator drafting followed by independent expert adjudication\.

### 3\.4Data Preprocessing

Raw entries are cleaned using regular expressions preserving Bangla Unicode \(U\+0980–U\+09FF\), reformatted into the instruction\-tuning template described in Section[4\.2](https://arxiv.org/html/2605.22487#S4.SS2), and partitioned into a 80% training set 3,356 examples 10% testing set 420 examples and 10% validation set 420 examples via stratified sampling across domain categories with a fixed random seed\. The complete pipeline is illustrated in Figure[1](https://arxiv.org/html/2605.22487#S4.F1)\.

## 4Methodology

![Refer to caption](https://arxiv.org/html/2605.22487v1/x1.png)Figure 1:BLADE methodology: three\-tier data collection, LoRA\-based fine\-tuning of pre\-trained multilingual LLMs, and a three\-way comparative evaluation frameworkHere we describe the experimental setups for both zero\-shot evaluation and supervised fine\-tuning\.

### 4\.1Zero\-Shot Setup

To establish a baseline for multilingual LLMs, we conducted a zero\-shot evaluation on our dataset, The dataset coverage all domain categories Table[3\.1](https://arxiv.org/html/2605.22487#S3.SS1)\. Each model was queried using the identical structured template described in Section[4\.2](https://arxiv.org/html/2605.22487#S4.SS2)to isolate true instruction\-following capability without task\-specific demonstrations\. It is important to note that zero\-shot models are evaluated on a prompt format they have not been explicitly trained on; consequently, some portion of the observed performance gap may reflectformat unfamiliarityrather than Bangla specific incapability alone\.

For inference, we utilized the Groq API111[https://console\.groq\.com/](https://console.groq.com/)to evaluateLlama\-4\-Scout\-17B,Gemma2\-9B, andKimi\-K2\-32B\. Google AI Studio was used to evaluateGemini\-2\.5\-FlashandGemini\-2\.0\-Flash\. Performance was benchmarked against BLADE ground\-truth responses\.

### 4\.2Fine\-tuning Setup

To adapt pre\-trained LLMs for Bangla instruction following, we employed Parameter\-Efficient Fine\-Tuning \(PEFT\) using 4\-bit NormalFloat \(NF4\) quantization and Low\-Rank Adaptation \(LoRA\)\(Huet al\.,[2021](https://arxiv.org/html/2605.22487#bib.bib24)\)\. We selected four architecturesDeepSeek\-R1\-Distill\-Llama\-8B,Qwen2\-1\.5B,Llama\-3\.2\-3B\-Instruct, andTigerLLM\-1B\-itto evaluate BLADE across varying scales and training paradigms\.TigerLLMserves as a Bangla\-native baseline; the remaining models represent state\-of\-the\-art multilingual architectures at the 1B–8B scale, chosen to test efficacy under resource\-constrained deployment conditions\.

All models were loaded in 4\-bit precision with FP16 computation via theUnslothlibrary on dual NVIDIA Tesla T4 GPUs \(32 GB total VRAM\)\. SFT was conducted for 2 epochs with a 2048 token sequence length usingSFTTrainer, batch size 2, cosine learning rate decay with 5 warm\-up steps at peak2×10−52\{\\times\}10^\{\-5\}, and AdamW optimizer\. LoRA was configured with rankr=16r\{=\}16,α=32\\alpha\{=\}32, dropout 0\.05, applied to both attention and MLP projection layers\. The prompt template have \(Listing[4\.2](https://arxiv.org/html/2605.22487#S4.SS2)\) enforced canonical document order \(Subject→\\rightarrowSalutation→\\rightarrowBody→\\rightarrowClosing\) and register consistency, with loss computed only over the<assistant\>span\.

Prompt Template<system\> order: Subject→\\rightarrowSalutation→\\rightarrowBody→\\rightarrowClosing\. Keep consistent honorifics\.<user\> Task:Formal application\. Subject: অগ্রিম ছুটির জন্য আবেদন Addressee: প্রধান শিক্ষক, এক্সওয়াইজেড উচ্চ বিদ্যালয় Applicant: মোঃ রহমান, নবম শ্রেণি, রোল ১২ Reason: ডাক্তার দেখাতে শহরে যেতে হবে Dates: ১০–১২ আষাঢ় ১৪৩২ Closing: বিনীতভাবে Constraints:150\-\-200 words;Bangla digits;nocode\-mixing\.<assistant\>

Table 4:Zero\-shot vs\. FM\(Format\-Matched\) performance across all models\. Subscript percentages indicate relative improvement of FM over Zero\-Shot \(↑\\uparrowhigher is better,↓\\downarrowlower is better for WER\)\. Best automatic metric scores inbold\. Human evaluation reports Structure / Fluency / Cultural Alignment respectively\.Table 5:Evaluation results before \(Base\) and after \(SFT\) fine\-tuning on BLADE\.Humanscores are averaged across Structure, Fluency, and Cultural Alignment \(1–5 scale\)\. LLM\-as\-judge ratings \(1–10 scale\) using GPT\-4\.1 across 450 samples\. Best SFT result per metric inbold\.
### 4\.3Evaluation Metrics

We evaluate model outputs using five complementary metrics that collectively capture lexical, structural, and semantic similarity between generated and reference texts\.BLEU\(Papineniet al\.,[2002](https://arxiv.org/html/2605.22487#bib.bib16)\)measuresnn\-gram overlap with a brevity penalty\.chrF\(Popović,[2015](https://arxiv.org/html/2605.22487#bib.bib13)\)computes a characternn\-gram F\-score, making it particularly robust for morphologically rich languages like Bangla\.ROUGE\-L\(Lin,[2004](https://arxiv.org/html/2605.22487#bib.bib17)\)measures the longest common subsequence between reference and hypothesis\.WER\(Ali and Renals,[2018](https://arxiv.org/html/2605.22487#bib.bib14)\)measures normalized edit distance between hypothesis and reference, where lower scores indicate fewer structural errors\.BERTScore\(Zhanget al\.,[2019](https://arxiv.org/html/2605.22487#bib.bib15)\)evaluates semantic similarity using contextual embeddings from a pretrained model\. Given the known limitations of similarity\-based metrics in capturing semantic coherence and cultural alignment, automatic metric results are complemented by human expert evaluation and LLM\-as\-judge scoring, both reported alongside automatic metrics in Section[5](https://arxiv.org/html/2605.22487#S5)\.

## 5Result Analysis

We benchmark our dataset through two assessments: \(1\) evaluating state\-of\-the\-art models in a zero\-shot setting and \(2\) comparing model performance before\(base model\) and after Supervised Fine\-Tuning \(SFT\)\. This dual approach highlights the dataset’s utility and the knowledge gap it addresses in NLP\.

### 5\.1Zero\-Shot and Format\-Matched Results

Table[4\.2](https://arxiv.org/html/2605.22487#S4.SS2)presents performance across five state\-of\-the\-art architectures under zero\-shot and format\-matched prompting\. Zero\-shot results confirm a significant pragmatic gap: no model exceeds 8 BLEU, with two systematic failure modes:structural displacement\(misplaced date, addressee, or closing blocks\) andhonorific mismatch\(inconsistentApni/Tumiregister\), as illustrated in Table[1](https://arxiv.org/html/2605.22487#S1.T1)\.

##### Zero\-Shot Performance\.

Gemini\-2\.5\-Flashachieves the strongest zero\-shot performance \(BLEU 7\.88, chrF 41\.04, BERTScore 0\.728\), yet confirms that even the strongest closed\-source baseline fails to produce functionally usable Bangla documents without targeted supervision\.Kimi\-K2\-32Bleads on ROUGE\-L \(0\.67\), though its lower BLEU reveals weak localnn\-gram precision\.LLaMA\-4\-Scout\-17Bachieves the lowest WER \(1\.73\), but this does not translate to overall generation quality, confirming that word\-level accuracy alone is insufficient for structured Bangla generation\.

##### Format\-Matched Prompting\.

Explicit structural constraints yield consistent improvements across all five models\.Gemini\-2\.5\-Flashcontinues to lead: BLEU 7\.88→\\rightarrow9\.14 \(↑\\uparrow16\.0%\), chrF 41\.04→\\rightarrow45\.67 \(↑\\uparrow11\.3%\), BERTScore 0\.728→\\rightarrow0\.751 \(↑\\uparrow3\.2%\), and WER 1\.934→\\rightarrow1\.203 \(↓\\downarrow37\.8%\) the largest absolute error reduction across all models\.Gemini\-2\.0\-Flashshows the most pronounced BLEU gain \(↑\\uparrow22\.8%\), whileGemma2\-9Brecords the largest ROUGE\-L gain \(↑\\uparrow34\.0%\), reflecting improved long\-range structural recall\.LLaMA\-4\-Scout\-17Bmaintains its WER advantage \(1\.521\), confirming stable word\-level alignment independent of prompting strategy\. BERTScore gains remain modest \(1\.8%–3\.2%\), indicating semantic fidelity is largely preserved in zero\-shot outputs and that format\-matching primarily addresses surface structural andnn\-gram precision failures\. These results confirm that explicit structural scaffolding is a lightweight yet effective intervention for closing the pragmatic gap in formal Bangla document generation, without requiring parameter updates or fine\-tuning\.

### 5\.2Fine\-tuning Results

Table[5](https://arxiv.org/html/2605.22487#S4.T5)presents automatic metrics, human evaluation, and LLM\-as\-judge scores before and after SFT across all four architectures\. The consistent gains across every model and every evaluation dimension confirm that BLADE provides a high\-fidelity, transferable training signal for low\-resource structured generation\.DeepSeek\-8Bexhibits the most pronounced improvement: a 22\-fold BLEU increase \(0\.78→\\rightarrow17\.73\), chrF rising from 9\.85 to 45\.87, and LLM\-judge score jumping from 3\.8 to 8\.9\. This delta reflects a fundamental shift in the model’s ability to handle Bangla’s inflectional morphology, structural syntax, and register consistency simultaneously\.Qwen2\-1\.5Bachieves the lowest post\-SFT WER \(0\.84\) and highest BERTScore \(0\.748\), indicating superior semantic alignment despite its compact size\. Its performance\-to\-parameter ratio establishes it as the strongest candidate for resource\-constrained deployment\.LLaMA3\.2\-3Battains the peak post\-SFT chrF \(46\.60\), confirming that targeted instruction\-tuning on BLADE allows smaller models to rival larger multilingual baselines on character\-level structural accuracy\.TigerLLM\-1B\-it, despite starting from the weakest baseline \(1\.20 BLEU, judge score 1\.5\), achieves a 3×\\timeschrF increase post\-SFT and a judge score of 7\.2 validating that BLADE’s expert\-curated supervision is effective even for Bangla\-native architectures that already possess foundational language knowledge\. Across all models, the average LLM\-judge score rises from 3\.15 to 8\.30, and human evaluation scores exceed 4\.6/5 on all three dimensions post\-SFT\. The strong alignment between automatic metrics, human judgment, and LLM\-judge scoring provides convergent validity for BLADE’s effectiveness as both a training resource and evaluation benchmark\.

#### 5\.2\.1Ablation Study

Table 6:Ablation summary table\. Deltas are averaged across models and computed relative to a vanilla SFT baseline \(same data, minimal prompt, max len 512, LoRAr=8r\{=\}8, linear LR, no label smoothing\)\.Abbrev\.:BL=BLEU, cF=chrF, WER=Word Error Rate,FmtTpl=format\-aware template,Roles=role tags,Ctx=context length,Pack=sliding\-window packing,LR=learning rate,LS=label smoothing,Attn=attention,MLP=feed\-forward block\. PositiveΔ\\Deltameans improvement; lower WER is better\.We investigate the drivers of these gains by fine\-tuning the base checkpoints on the BLADE training split\. Metrics are averaged across models\. Our final configuration \(Table[6](https://arxiv.org/html/2605.22487#S5.T6)\) uses: AdamW \(LR2×10−52\{\\times\}10^\{\-5\}, cosine decay, 3% warmup\), batch size 2 \(gradient accumulation\), sequence length 2048 with packing, LoRAr=16r\{=\}16\(α=32\\alpha\{=\}32\) on attention/MLP, label smoothing 0\.1, and a format\-aware template with role tags\.

Training on larger fractions of BLADE \(25%–100%\) yields monotonic gains, with the steepest jump at 50% \(avg\.Δ\\DeltaBLEU \+5\.4;Δ\\DeltachrF \+9\.1\)\. The full set delivers the lowest WER, underscoring that targeted supervision drives structural fidelity\. Prompt design is equally critical: a format\-aware template surfacing fields likeSubjectsubstantially reduces WER \(−0\.18\-0\.18\) and boosts ROUGE\-L \(\+0\.08\), while explicit role tags further improve BERTScore and chrF \(\+2\.7\)\. Context length is decisive; truncating to 512 tokens degrades long\-form fidelity, and 1024 tokens often miss signatures given the average response length of 1,300\. Extending to 2048 with sliding\-window packing ensures complete structure, yielding the strongest structural scores \(avg\.Δ\\DeltachrF \+3\.5\)\. Adapter capacity also matters: LoRAr=16r\{=\}16outperformsr=8r\{=\}8\(avg\.Δ\\DeltaBLEU \+1\.3\) without the overfitting seen atr=32r\{=\}32\. Cosine decay with a peak LR of2×10−52\{\\times\}10^\{\-5\}proved robust across architectures, whereas higher rates degraded stability\. Label smoothing \(0\.1\) improves semantic alignment \(avg\.Δ\\DeltaBS \+0\.006\) and mitigates brittle copying\. Applying LoRA to both attention*and*MLP projections beats attention\-only adaptation \(avg\.Δ\\DeltaBLEU \+0\.9;Δ\\DeltachrF \+1\.8\)\. Finally, mixed precision \(bf16/fp16\) matches full\-precision quality while enabling larger batches and slightly improving WER \(−0\.03\-0\.03\) via stable optimization\.

### 5\.3LLM\-as\-Judge Procedure

To assess functional usability beyond automatic metrics, we employedGPT\-4\.1as an LLM judge across 450 randomized samples\. The judge was provided with the input prompt \(Appendix[B](https://arxiv.org/html/2605.22487#A2)\), the rationale for selectingGPT\-4\.1\(Appendix[B\.2](https://arxiv.org/html/2605.22487#A2.SS2)\), a reference response, and the model output, and instructed to score on a 1–10 scale against three criteria: \(i\)structural correctnesspresence and ordering of all mandatory document components; \(ii\)register consistencyhonorific alignment throughout; and \(iii\)semantic relevancewhether the output addresses the stated prompt purpose\. The judge prompt was fixed across all 450 samples to ensure scoring consistency, and outputs were parsed programmatically; malformed scores were re\-queried once before exclusion\.

### 5\.4Human Evaluation

Three native Bangla\-speaking philology experts conducted a double\-blind assessment of 100 stratified samples, rating outputs on a 5\-point Likert scale across: \(i\)Structural Integrityadherence to canonical document formatting; \(ii\)Fluencygrammatical correctness and naturalness; and \(iii\)Cultural Alignmenthonorific register consistency\. Inter\-annotator agreement was robust \(Spearman’sρ=0\.84\\rho\{=\}0\.84, Kendall’sτ=0\.76\\tau\{=\}0\.76\), and post\-SFT scores exceeded 4\.6/5 across all dimensions \(Table[5](https://arxiv.org/html/2605.22487#S4.T5)\), confirming that BLADE fine\-tuning enables models to produce institutionally appropriate, functionally usable Bangla documents\.

## 6Discussion

The central finding of BLADE is straightforward: for low\-resource languages like Bangla, cultural specificity of instruction\-tuning data matters more than model scale\.Qwen2\-1\.5Bfine\-tuned on BLADE achieves BERTScore 0\.748 and WER 0\.84, outperforming zero\-shotGemini\-2\.5\-Flashon both metrics despite being orders of magnitude smaller confirming that expert\-curated, register\-aware supervision unlocks capabilities that general\-purpose pretraining at any scale cannot substitute\.

##### Qualitative Pragmatic Competence Shift

Zero\-shot outputs fail in two institutionally fatal ways:structural displacement\(missing or misplaced date blocks, subject lines, or closings\) andhonorific mismatch, where models oscillate between formal আপনি \(Apni\) and informal তুমি \(Tumi\) registers within a single document \(Table[1](https://arxiv.org/html/2605.22487#S1.T1)\)\. No institution would accept such output the strongest zero\-shot model scores only 1\.85/5 on Structural Integrity\. Post\-SFT, both failure modes are eliminated: models produce correctly formatted date and addressee blocks, maintain formal register across all verb conjugations, and close with culturally appropriate markers such as বিনীতভাবে জানাচ্ছি \(I respectfully inform you\), representing a qualitative shift from outputs thatappearfluent to outputs that arefunctionally usable\. chrF captures this most faithfully: sensitive to the inflectional suffixes and agglutinative morphemes carrying honorific information in Bangla, a jump from 9\.85 to 45\.87 forDeepSeek\-8Breflects correct conjugations, consistent honorific suffixes, and structurally complete documents explaining its strongest correlation with human expert scores\.

##### Validation Across Evaluation Dimensions

Three independent evaluation channels human philologists operating double\-blind, an LLM judge, and reference\-based automatic metrics converge on the same conclusion\. Human evaluators awarded post\-SFT outputs above 4\.6/5 across all dimensions, with Structural Integrity showing the largest gain \(1\.85→\\rightarrow4\.72\)\. The LLM\-judge average rose from 3\.15 to 8\.30 across 450 samples, tracking quantitative deltas precisely:DeepSeek\-8B’s 22\-fold BLEU increase corresponds to the highest judge score \(8\.9\),LLaMA3\.2\-3B’s peak chrF of 46\.60 to a score of 8\.7, andTigerLLM\-1B’s modest gains to the lowest post\-SFT score \(7\.2\)\. This convergence rules out metric\-specific artifacts and confirms that BLADE produces genuine improvement in functional usability the exact quality that zero\-shot multilingual pretraining systematically fails to provide\.

## 7Conclusion

We presentedBLADE, a 4,196\-pair instruction\-tuning dataset encoding the structural, register, and cultural conventions for usable Bangla generation\. While baseline multilingual models fell short, supervised fine\-tuning on BLADE delivered large, consistent gains across architecturesincluding both multilingual and Bangla\-native models, converting surface\-level fluency into outputs meeting real\-world expectations\. Our findings confirm that carefully curated, domain\-specific supervision unlocks capabilities that generic pretraining and parameter count cannot, enabling compact models to rival larger ones on practical tasks\. We aim for BLADE to catalyze structure\-aware, register\-sensitive generation in Bangla and other underserved languages, encouraging evaluations that pair automatic metrics with format checks and human judgments, with future extensions to additional genres and integrated validators\.

## Limitations

Our study has several constraints that present opportunities for future refinement\.

First, all models were fine\-tuned for only two epochs due to hardware constraints \(dual NVIDIA Tesla T4 GPUs, 32 GB total VRAM\), which may have limited convergence and overall model stability\. These same resource constraints precluded exhaustive hyperparameter search and more comprehensive ablation studies across a wider range of configurations\. Second, the BLADE dataset is currently biased toward formal Bangla registers, as informal and dialectal sources were less accessible during collection\. This limits the generalizability of fine\-tuned models to informal conversational contexts\.

Future work will address these limitations by extending training duration with more capable hardware, expanding dataset coverage to include informal Bangla dialects and regional varieties, and conducting broader evaluations across additional document genres and task types\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.Cited by:[§1](https://arxiv.org/html/2605.22487#S1.p1.1)\.
- Word error rate estimation for speech recognition: e\-WER\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),I\. Gurevych and Y\. Miyao \(Eds\.\),Melbourne, Australia,pp\. 20–24\.External Links:[Link](https://aclanthology.org/P18-2004/),[Document](https://dx.doi.org/10.18653/v1/P18-2004)Cited by:[§4\.3](https://arxiv.org/html/2605.22487#S4.SS3.p1.2)\.
- M\. Attia, A\. Muhamed, M\. Alkhamissi, T\. Solorio, and M\. T\. Diab \(2026\)Beyond understanding: evaluating the pragmatic gap in LLMs’ cultural processing of figurative language\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 7238–7265\.External Links:[Link](https://aclanthology.org/2026.eacl-long.341/),[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.341),ISBN 979\-8\-89176\-380\-7Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Bawden and F\. Yvon \(2023\)Investigating the translation performance of a large multilingual language model: the case of BLOOM\.InProceedings of the 24th Annual Conference of the European Association for Machine Translation,M\. Nurminen, J\. Brenner, M\. Koponen, S\. Latomaa, M\. Mikhailov, F\. Schierl, T\. Ranasinghe, E\. Vanmassenhove, S\. A\. Vidal, N\. Aranberri, M\. Nunziatini, C\. P\. Escartín, M\. Forcada, M\. Popovic, C\. Scarton, and H\. Moniz \(Eds\.\),Tampere, Finland,pp\. 157–170\.External Links:[Link](https://aclanthology.org/2023.eamt-1.16/)Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Bhattacharjee, T\. Hasan, W\. Ahmad, K\. S\. Mubasshir, M\. S\. Islam, A\. Iqbal, M\. S\. Rahman, and R\. Shahriyar \(2022\)BanglaBERT: language model pretraining and benchmarks for low\-resource language understanding evaluation in Bangla\.InFindings of the Association for Computational Linguistics: NAACL 2022,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 1318–1327\.External Links:[Link](https://aclanthology.org/2022.findings-naacl.98/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-naacl.98)Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Cahyawijaya, H\. Lovenia, and P\. Fung \(2024\)LLMs are few\-shot in\-context low\-resource language learners\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 405–433\.External Links:[Link](https://aclanthology.org/2024.naacl-long.24/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.24)Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px1.p1.1)\.
- O\. de Gibert, G\. Nail, N\. Arefyev, M\. Bañón, J\. van der Linde, S\. Ji, J\. Zaragoza\-Bernabeu, M\. Aulamo, G\. Ramírez\-Sánchez, A\. Kutuzov, S\. Pyysalo, S\. Oepen, and J\. Tiedemann \(2024\)A new massive multilingual dataset for high\-performance language technologies\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 1116–1128\.External Links:[Link](https://aclanthology.org/2024.lrec-main.100/)Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px1.p1.1)\.
- M\. R\. Farhansyah, I\. Darmawan, A\. Kusumawardhana, G\. I\. Winata, A\. F\. Aji, and D\. T\. Wijaya \(2025\)Do language models understand honorific systems in Javanese?\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 26732–26754\.External Links:[Link](https://aclanthology.org/2025.acl-long.1296/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1296),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2021\)LoRA: low\-rank adaptation of large language models\.External Links:[Link](https://arxiv.org/abs/2106.09685)Cited by:[§4\.2](https://arxiv.org/html/2605.22487#S4.SS2.p1.1)\.
- K\. I\. Islam, S\. Kar, M\. S\. Islam, and M\. R\. Amin \(2021\)SentNoB: a dataset for analysing sentiment on noisy Bangla texts\.InFindings of the Association for Computational Linguistics: EMNLP 2021,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Punta Cana, Dominican Republic,pp\. 3265–3271\.External Links:[Link](https://aclanthology.org/2021.findings-emnlp.278/),[Document](https://dx.doi.org/10.18653/v1/2021.findings-emnlp.278)Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px3.p1.1)\.
- V\. Iyer, B\. Malik, P\. Stepachev, P\. Chen, B\. Haddow, and A\. Birch \(2024\)Quality or quantity? on data scale and diversity in adapting large language models for low\-resource translation\.InProceedings of the Ninth Conference on Machine Translation,B\. Haddow, T\. Kocmi, P\. Koehn, and C\. Monz \(Eds\.\),Miami, Florida, USA,pp\. 1393–1409\.External Links:[Link](https://aclanthology.org/2024.wmt-1.128/),[Document](https://dx.doi.org/10.18653/v1/2024.wmt-1.128)Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px4.p1.1)\.
- J\. R\. Landis and G\. G\. Koch \(1977\)The measurement of observer agreement for categorical data\.pp\. 159–174\.Cited by:[§A\.4](https://arxiv.org/html/2605.22487#A1.SS4.p1.2)\.
- C\. Lin \(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,Barcelona, Spain,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§4\.3](https://arxiv.org/html/2605.22487#S4.SS3.p1.2)\.
- A\. Mhaske, H\. Kedia, S\. Doddapaneni, M\. M\. Khapra, P\. Kumar, R\. Murthy, and A\. Kunchukuttan \(2023\)Naamapadam: a large\-scale named entity annotated data for Indic languages\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 10441–10456\.External Links:[Link](https://aclanthology.org/2023.acl-long.582/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.582)Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Mukherjee, A\. Mehta, S\. Saha, A\. Arora, and M\. Choudhury \(2025\)Women, infamous, and exotic beings: a comparative study of honorific usages in Wikipedia and LLMs for Bengali and Hindi\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 19103–19126\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.966/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.966),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2605.22487#S1.p1.1)\.
- S\. K\. Nahin, R\. N\. Nandi, S\. Sarker, Q\. S\. Muhtaseem, M\. Kowsher, A\. C\. Shill, M\. Ibrahim, M\. H\. Menon, T\. A\. Muntasir, and F\. Alam \(2025\)TituLLMs: a family of Bangla LLMs with comprehensive benchmarking\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 24922–24940\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1279/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1279),ISBN 979\-8\-89176\-256\-5Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu \(2002\)Bleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,P\. Isabelle, E\. Charniak, and D\. Lin \(Eds\.\),Philadelphia, Pennsylvania, USA,pp\. 311–318\.External Links:[Link](https://aclanthology.org/P02-1040/),[Document](https://dx.doi.org/10.3115/1073083.1073135)Cited by:[§4\.3](https://arxiv.org/html/2605.22487#S4.SS3.p1.2)\.
- B\. Paul, F\. F\. Preotee, S\. Sarker, S\. R\. Refat, S\. Islam, T\. Muhammad, M\. A\. Hoque, and S\. Manzoor \(2026\)ANCHOLIK\-ner: a benchmark dataset for bangla regional named entity recognition\.PLOS ONEProcedia Computer SciencearXiv e\-printsURL https://arxiv\. org/abs/2407\.10671arXiv preprint arXiv:2501\.12948ICLRarXiv preprint arXiv:1904\.09675arXiv preprint arXiv:2303\.08774arXiv preprint arXiv:2312\.11805Journal of Modern LanguagesarXiv preprint arXiv:2106\.09685biometrics21,pp\. 1–36\.External Links:[Document](https://dx.doi.org/10.1371/journal.pone.0342786),[Link](https://doi.org/10.1371/journal.pone.0342786)Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Popović \(2015\)ChrF: character n\-gram F\-score for automatic MT evaluation\.InProceedings of the Tenth Workshop on Statistical Machine Translation,O\. Bojar, R\. Chatterjee, C\. Federmann, B\. Haddow, C\. Hokamp, M\. Huck, V\. Logacheva, and P\. Pecina \(Eds\.\),Lisbon, Portugal,pp\. 392–395\.External Links:[Link](https://aclanthology.org/W15-3049/),[Document](https://dx.doi.org/10.18653/v1/W15-3049)Cited by:[§4\.3](https://arxiv.org/html/2605.22487#S4.SS3.p1.2)\.
- M\. Rahman, N\. Mohammed, N\. Mansoor, and S\. Momen \(2019\)Chittron: an automatic bangla image captioning system\.154,pp\. 636–642\.Note:Proceedings of the 9th International Conference of Information and Communication Technology \[ICICT\-2019\] Nanning, Guangxi, China January 11\-13, 2019External Links:ISSN 1877\-0509,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.procs.2019.06.100),[Link](https://www.sciencedirect.com/science/article/pii/S1877050919308701)Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px3.p1.1)\.
- N\. Raihan and M\. Zampieri \(2025\)TigerLLM \- a family of Bangla large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 887–896\.External Links:[Link](https://aclanthology.org/2025.acl-short.69/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-short.69),ISBN 979\-8\-89176\-252\-7Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px3.p1.1)\.
- R\. Sekizawa and H\. Yanaka \(2023\)Analyzing syntactic generalization capacity of pre\-trained language models on Japanese honorific conversion\.InProceedings of the 12th Joint Conference on Lexical and Computational Semantics \(\*SEM 2023\),A\. Palmer and J\. Camacho\-collados \(Eds\.\),Toronto, Canada,pp\. 40–47\.External Links:[Link](https://aclanthology.org/2023.starsem-1.5/),[Document](https://dx.doi.org/10.18653/v1/2023.starsem-1.5)Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px2.p1.1)\.
- R\. Snigdha and M\. Rahman \(2022\)Representation of social class and hierarchy in Bangla address terms: a sociolinguistic study\.32\(2\),pp\. 107–129\.External Links:[Link](https://jml.um.edu.my/index.php/JML/article/view/39438)Cited by:[§1](https://arxiv.org/html/2605.22487#S1.p1.1),[§1](https://arxiv.org/html/2605.22487#S1.p2.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. C\. Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. Scialom \(2023\)Llama 2: open foundation and fine\-tuned chat models\.External Links:2307\.09288,[Link](https://arxiv.org/abs/2307.09288)Cited by:[§1](https://arxiv.org/html/2605.22487#S1.p1.1)\.
- B\. Zeng, C\. Lyu, S\. Liu, M\. Zeng, M\. Wu, X\. Ni, T\. Shi, Y\. Zhao, Y\. Liu, C\. Zhu, R\. Li, J\. Geng, Q\. Li, Y\. Tong, L\. Wang, W\. Luo, and K\. Zhang \(2025\)Marco\-bench\-MIF: on multilingual instruction\-following capability of large language\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 24058–24072\.External Links:[Link](https://aclanthology.org/2025.acl-long.1172/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1172),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi \(2019\)Bertscore: evaluating text generation with bert\.Cited by:[§4\.3](https://arxiv.org/html/2605.22487#S4.SS3.p1.2)\.
- C\. Zhou, P\. Liu, P\. Xu, S\. Iyer, J\. Sun, Y\. Mao, X\. Ma, A\. Efrat, P\. Yu, L\. YU, S\. Zhang, G\. Ghosh, M\. Lewis, L\. Zettlemoyer, and O\. Levy \(2023\)LIMA: less is more for alignment\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 55006–55021\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/ac662d74829e4407ce1d126477f4a03a-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2605.22487#S2.SS0.SSS0.Px4.p1.1)\.

## Appendix AAnnotation Protocol and Quality Control

### A\.1Annotator Profile

The dataset was constructed and validated by seven annotators: five paper authors with backgrounds in computer science and Bangla linguistics, and two external native linguistic experts specializing in Bangla philology and sociolinguistics\. All annotators are native Bangla speakers with formal education conducted in Bangla\. External experts were compensated at a standard research rate, informed of the research purpose prior to participation, and provided explicit written consent for inclusion\.

### A\.2Annotation Guidelines

Annotators followed a written guideline document evaluating each entry across three dimensions:

##### 1\. Structural Completeness\.

Structural completeness is a binary pass/fail criterion evaluated against a document\-type\-specific checklist\. A response is structurally complete if and only if it containsallmandatory components for its document type in the correct canonical order\. For formal applications, the seven mandatory components are:

1. 1\.Date in Bangla format \(e\.g\., ১৮/১০/২০২৪ খ্রিঃ\)
2. 2\.Addressee block: recipient name/title, institution, and address
3. 3\.Subject line \(বিষয়ঃ\)
4. 4\.Formal salutation \(মহোদয় or equivalent\)
5. 5\.Body: minimum two paragraphs — context statement and formal request
6. 6\.Formal closing \(বিনীত or equivalent\)
7. 7\.Applicant signature block: name, class/position, roll/ID, institution

A response missing any component, or presenting components out of canonical order, fails this criterion and is either corrected \(minor errors, e\.g\., missing date field\) or discarded \(structural malformation or register inconsistency\)\.

##### 2\. Honorific Consistency\.

A response passes honorific consistency if all second\-person pronouns, associated verb forms, and relational terms maintain a single register throughout\. Formal register requires exclusive use of আপনি \(Apni: you\) and its associated verb conjugations\. Any occurrence of informal forms তুমি \(Tumi: you\) or তুই \(Tui: you\) within a formally\-labeled entry constitutes an honorific violation and triggers rejection or correction\. For dialogue entries, the required register \(formal/informal\) is determined by the topic label and must remain consistent throughout all turns\.

##### 3\. Cultural and Contextual Accuracy\.

Content must reflect realistic Bangladeshi institutional contexts: plausible institution names, dates in correct Bangla calendar or AD format, and discourse markers appropriate to the document type \(e\.g\., অতএব for formal petition closings, বিনীতভাবে জানাচ্ছি for formal body openings\)\.

### A\.3Annotation Workflow

##### Tier 1 and Tier 2 entries

underwent two\-annotator verification against all three criteria\. A stratified random sample of 15% of these entries 630 examples was independently re\-verified by a second annotator to estimate verification reliability\.

##### Tier 3 entries

followed a stricter two\-stage workflow: \(i\) draft production by one author\-annotator, and \(ii\) independent review by a second annotator against all three criteria\. Disagreements were escalated to one of the two external linguistic experts for final adjudication\. No Tier 3 entry was included without passing both stages\.

### A\.4Inter\-Annotator Agreement

Cross\-validation on the 630\-entry random sample yielded Cohen’sκ=0\.81\\kappa=0\.81for structural completeness judgments andκ=0\.79\\kappa=0\.79for honorific consistency judgments, indicating strong inter\-annotator agreement\(Landis and Koch,[1977](https://arxiv.org/html/2605.22487#bib.bib25)\)\. Disagreements on the cross\-validation sample were resolved by majority vote among three annotators\.

### A\.5Quality Control Mechanisms

Three dataset\-level quality control mechanisms were applied following per\-entry annotation:

##### Automated Register Audit\.

All 4,196 entries underwent automated pre\-screening using regular expression filters to flag occurrences of informal pronoun forms তুমি, তুই, তোমার, তোর means you, within formally\-labeled entries\. Flagged entries were manually reviewed; confirmed violations were corrected or discarded\.

##### Structural Completeness Audit\.

A secondary pass verified that all seven mandatory components were present for every formal application entry, using the checklist defined in Section[A](https://arxiv.org/html/2605.22487#A1)\.2\. Entries failing this audit after the initial annotation phase were returned to annotators for correction\.

##### Semantic Diversity Check\.

To prevent topic redundancy across 2,008 topics, character n\-gram similarity was computed across all topic labels\. Topic pairs with similarity exceeding 0\.85 were flagged for manual review; confirmed near\-duplicates were merged or one was discarded, ensuring each topic represents a genuinely distinct writing scenario\.

## Appendix BMore Details on LLM\-as\-Judge

### B\.1Prompt Template

LLM\-as\-Judge Evaluation PromptYou are evaluating a Bangla text generation model\. Input Prompt:\{prompt\} Reference Response:\{reference\} Model Output:\{output\} Score the model output from 1\-\-10 based on: \(i\)Structural Correctness: Are all mandatory document components present in canonical order? \(ii\)Register Consistency: Are honorifics \(Apni/Tumi:you\) used consistently throughout? \(iii\)Semantic Relevance: Does the output address the stated prompt purpose? Return only a single integer score between 1 and 10\.

### B\.2Rationale for Selecting GPT\-4\.1 as LLM Judge

AS our zero\-shot evaluation included two Gemini\-family models, Gemini\-2\.0\-Flash and Gemini\-2\.5\-Flash as evaluated systems, we selected GPT\-4\.1 as the LLM judge to avoid self\-evaluation bias, a well\-documented confound in LLM\-as\-judge frameworks whereby a model tends to favor outputs stylistically similar to its own generations\. GPT\-4\.1 was chosen over alternative judge candidates for two reasons\. First, its instruction\-following reliability and structured output compliance made it well\-suited for the fixed rubric scoring task, with fewer than 2% of judge responses requiring re\-querying due to malformed outputs\. Second, its strong multilingual comprehension, particularly for morphologically rich languages, made it capable of detecting the honorific inconsistencies and structural violations central to our evaluation rubric qualities that a purely English\-centric judge would likely miss\.

Table 7:Model Output Before and After BLADE Fine\-Tuning with LLaMA 3\.2
Before Fine\-tuning \(English\)After Fine\-tuning \(English\)Prompt:Application for advance leavePriva, usually 15\-17\.02\.2023 \| <\|which phone attempt addon?\|\>Happiness: High centerEvolution: Style’s dense blossomSexual casualties can burn strictly\.Provision:\(1\) \(2\)\(1\)\(0\)China ChinaBalance:−20∘\-20^\{\\circ\}C/−4∘\-4^\{\\circ\}F \(\-32F to \-19F\)Voucher:−60∘\-60^\{\\circ\}C/−76∘\-76^\{\\circ\}FHow many times to sit for mother\-mantra before application?\* Daily\> \* One Daily \*Answer:\> \* Every unity hard jam\-burn \*Computing graphic bandingHappiness: High centerEvolution: Style’s dense blossomSexual casualties can burn strictly\.Provision:\(1\) \(2\)\(1\)\(0\)China ChinaBalance:−80∘\-80^\{\\circ\}C/\-113F\(\-112F\)−150∘\-150^\{\\circ\}C/\-238FPrompt:Application for advance leaveDate: 18/10/2024 ADToHeadmasterSolmaid High School and CollegeVatara, Dhaka\-1212Subject:Application to the Headmaster for advance leave\.Sir,With due respect, I state that I am a regular student of class nine in your school\. I need advance leave from 21/10/2024 to 23/10/2024 on the occasion of my elder sister’s marriage\. I have to help the elders in the said ceremony, so I will not be able to attend school\.Therefore, a fervent appeal to the Sir, that you would be kind enough to grant me the mentioned three days of leave\.Sincerely,Your most obedient student\(Your Name\)Class\-Nine, Roll\-12Department\-ScienceSolmaid High School and CollegeVatara, Dhaka\-1212Humble appeal to Sir for the matter, requesting your overall well\-being that you consider my advance leave application humanely\.Table 8:English\-translation of Model Output Before and After BLADE Fine\-Tuning with LLaMA 3\.2\.Table 9:List of Source Websites & Books for Bangla Language ContentTable 10:Comparison of LLaMA 3\.2 Model Performance Before and After Fine\-tuning