SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for Low-Resource Nepali Financial Speech Recognition

# Domain-Adaptive LoRA Fine-Tuning of Whisper for Low-Resource Nepali Financial Speech Recognition
Source: [https://arxiv.org/html/2609.01737](https://arxiv.org/html/2609.01737)
September 2026

###### Abstract

Mobile payment applications in Nepal are graphically mediated and largely inaccessible to visually impaired users\. This paper presents SpeakPay, a voice\-first digital wallet, and documents the central technical contribution: a controlled study of domain adaptation for low\-resource financial speech recognition\. We introduce NepFinSpeech\-403, a 403\-utterance dataset of Nepali financial voice commands \(send, load, and balance operations spanning 237 unique numerals\), and fine\-tune Whisper large\-v2 with LoRA\. On the held\-out test set, the domain\-adapted model reduces Word Error Rate from 129\.95% \(zero\-shot baseline\) to 42\.58% — a 67\.2% relative reduction — and improves Devanagari numeral recognition accuracy from 0\.0% to 73\.9%\. We find that word\-level metrics understate the practical task\-level impact: domain adaptation improves the Transaction Success Rate from 1\.67% to 33\.33%, a roughly 20×\\timesgain\. The improvement is consistent at the individual\-utterance level \(sign test,p<10−17p<10^\{\-17\}\) and across all command types\. A data efficiency analysis shows that as few as 100 domain\-specific utterances are sufficient to halve the zero\-shot WER, with performance plateauing around 300 examples\. Error analysis reveals systematic numeral confusion patterns \(zero insertion/deletion, prefix hallucination\) that account for the majority of remaining transaction failures\. The trained system is deployed as a publicly accessible voice\-first web application\. All code, dataset, model weights, and this paper are released at[https://github\.com/subedibiraj/speakpay](https://github.com/subedibiraj/speakpay)\.

Keywords:Automatic Speech Recognition, Low\-Resource Languages, Nepali, Domain Adaptation, LoRA, Whisper, Financial Speech, Accessibility

## 1\. Introduction

Mobile wallets \(eWallets\) such as eSewa and Khalti have become the dominant mode of digital payment in Nepal\. Their interfaces, however, are built entirely around visual interaction — icon\-based navigation, on\-screen forms, and graphical confirmation flows\. For Nepal’s visually impaired population \(estimated at roughly 95,000 individuals\[[1](https://arxiv.org/html/2609.01737#bib.bib6)\]\), this makes independent use of digital finance effectively impossible without sighted assistance\.

Voice interaction is the natural accessibility solution, but two gaps stand in the way\. First, existing Nepali Automatic Speech Recognition \(ASR\) systems are trained and evaluated on general speech\[[2](https://arxiv.org/html/2609.01737#bib.bib3),[3](https://arxiv.org/html/2609.01737#bib.bib4)\], not financial commands, which carry domain\-specific demands: dense numeral sequences \(transfer amounts, account balances\), proper nouns \(bank names, recipient names\), and a narrow set of recurring sentence structures\. Second, no public Nepali financial speech dataset existed prior to this work, making it impossible to even measure how badly general\-purpose models perform on this domain, let alone improve on it\.

This paper documents the construction of such a dataset and a controlled comparison of zero\-shot and domain\-adapted ASR on it\. The project began as an undergraduate group project with no rigorous evaluation, no public dataset, and no reproducible benchmark\. This version is a complete independent rewrite: a newly assembled and characterized dataset, a reproducible LoRA fine\-tuning pipeline, a statistically grounded evaluation, and a deployed application\.

Our contributions are:

1. 1\.NepFinSpeech\-403, the first public Nepali financial speech dataset, released on Hugging Face under CC\-BY 4\.0\.
2. 2\.A controlled comparison showing that LoRA fine\-tuning reduces WER by 67\.2% relative to the zero\-shot baseline and improves the Transaction Success Rate from 1\.67% to 33\.33%\.
3. 3\.A data efficiency analysis demonstrating that 100–300 domain\-specific utterances suffice for most of the available adaptation gain\.
4. 4\.An error taxonomy of Devanagari numeral confusions \(zero insertion/deletion, prefix hallucination\) specific to the financial speech domain\.
5. 5\.A deployed, publicly accessible voice\-first eWallet application integrating the adapted ASR model\.

## 2\. Related Work

Nepali ASR\.Regmi and Bal\[[2](https://arxiv.org/html/2609.01737#bib.bib3)\]built an end\-to\-end Nepali ASR system with ESPnet on the OpenSLR\-54 corpus\[[4](https://arxiv.org/html/2609.01737#bib.bib18)\], reporting a Character Error Rate \(CER\) of 10\.3% on 159k general\-domain utterances\. Paudel et al\.\[[3](https://arxiv.org/html/2609.01737#bib.bib4)\]used a CNN\-Transformer architecture, achieving CER of 11\.14% on a similarly general corpus\. Both systems are evaluated on read speech or broad\-domain utterances; neither reports performance on financial or transactional language, which differs substantially in vocabulary distribution and numeral density\.

Multilingual and low\-resource speech recognition\.Self\-supervised pretraining\[[5](https://arxiv.org/html/2609.01737#bib.bib14)\]and large\-scale weakly supervised training\[[6](https://arxiv.org/html/2609.01737#bib.bib1)\]have substantially improved ASR for low\-resource languages\. The FLEURS benchmark\[[7](https://arxiv.org/html/2609.01737#bib.bib12)\]provides standardized evaluation across 102 languages including Nepali, but measures general\-domain performance only\. Meta’s Massively Multilingual Speech project\[[8](https://arxiv.org/html/2609.01737#bib.bib13)\]extended coverage to over 1,000 languages, and IndicSUPERB\[[9](https://arxiv.org/html/2609.01737#bib.bib17)\]benchmarked speech tasks across Indian languages\. These efforts confirm that domain mismatch remains a persistent challenge even when general\-domain coverage improves\. Distillation approaches like Distil\-Whisper\[[10](https://arxiv.org/html/2609.01737#bib.bib15)\]reduce model size but do not address domain gaps\. Our work complements these efforts by measuring the domain gap for a specific high\-stakes application\.

Parameter\-efficient fine\-tuning\.Hu et al\.\[[11](https://arxiv.org/html/2609.01737#bib.bib2)\]introduced LoRA, which freezes the base model and injects trainable low\-rank decomposition matrices into attention and feed\-forward layers, reducing trainable parameters by over 99% relative to full fine\-tuning while matching or exceeding its quality on many tasks\. Dettmers et al\.\[[12](https://arxiv.org/html/2609.01737#bib.bib11)\]extended this with quantization \(QLoRA\), further reducing memory requirements\. Recent work has compared LoRA, full fine\-tuning, and prompt tuning for domain\-specific Whisper adaptation\[[13](https://arxiv.org/html/2609.01737#bib.bib16)\], generally finding LoRA competitive with full fine\-tuning at a fraction of the cost\. These methods make adaptation feasible on a single consumer GPU and on a dataset far too small to safely full\-fine\-tune a 1\.55\-billion\-parameter model\.

Task\-level ASR evaluation\.Standard WER treats all word errors equally, but downstream task performance can diverge sharply from word\-level accuracy\. Kim et al\.\[[14](https://arxiv.org/html/2609.01737#bib.bib7)\]proposed Semantic Distance as an alternative metric that better predicts spoken language understanding performance\. Fu et al\.\[[15](https://arxiv.org/html/2609.01737#bib.bib8)\]defined Interpretation Error Rate \(IRER\) as an utterance\-level, no\-partial\-credit metric for joint intent and slot correctness\. We adopt a variant of IRER \(reported as Transaction Success Rate\) to evaluate whether transcription errors actually cause transaction failures\.

Accessible FinTech\.Concurrent work on accessible voice payments\[[16](https://arxiv.org/html/2609.01737#bib.bib9),[17](https://arxiv.org/html/2609.01737#bib.bib10)\]has focused on system architecture, USSD automation, and biometric security for visually impaired users\. We focus instead on the ASR domain\-adaptation and evaluation\-metric questions underlying the language layer these systems depend on, using Nepali — a language with no existing accessible financial voice interface — as a case study\.

Our work does not propose a new architecture or training method\. We study whether domain\-specific LoRA adaptation closes the gap between a general\-purpose ASR model and the requirements of financial voice interaction in a low\-resource language, and deploy the result as a working application\.

## 3\. Dataset: NepFinSpeech\-403

### 3\.1 Collection

Audio was collected from undergraduate students and staff at Advanced College of Engineering and Management \(Tribhuvan University\) through a purpose\-built web platform \([https://bolanepal\.netlify\.app](https://bolanepal.netlify.app/)\)\. Contributors recorded spoken Nepali financial commands guided by written prompts, covering three operation types: fund transfers \(recipient, amount, optionally a financial institution\), wallet load/deposit commands, and balance enquiries\. Recordings were captured via browser microphone \(varied devices, no controlled acoustic environment\) and stored as WAV files at 16 kHz\. Transcripts were manually verified against the recorded audio\. Contributor identity was not recorded per utterance during collection, so speaker overlap between train and test splits cannot be entirely ruled out\.

### 3\.2 Statistics

Table 1:NepFinSpeech\-403 composition
The “other” category is a heuristic residual class assigned by keyword matching, not a manually curated label; Section[6](https://arxiv.org/html/2609.01737#S6)shows it captures a genuine, previously unaccounted\-for intent \(third\-person “funds received” statements\), which we treat as a finding rather than noise\.

## 4\. Method

Base model\.Whisper large\-v2\[[6](https://arxiv.org/html/2609.01737#bib.bib1)\], 1\.55B parameters, encoder\-decoder Transformer operating on log\-Mel spectrograms\.

Adaptation\.LoRA\[[11](https://arxiv.org/html/2609.01737#bib.bib2)\]with rankr=32r=32, scalingα=64\\alpha=64, applied to all query, key, value, output, and feed\-forward projection matrices \(q\_proj,k\_proj,v\_proj,out\_proj,fc1,fc2\) across every encoder and decoder layer\. This yields approximately 8M trainable parameters, under 0\.6% of the base model, while the remaining 99\.4% stays frozen\.

Why LoRA, not full fine\-tuning\.With only 303 training utterances, full fine\-tuning of a 1\.55B\-parameter model risks both overfitting and catastrophic forgetting of the base model’s general speech competence\. LoRA’s parameter efficiency is not merely a compute convenience here — it is the difference between a feasible and an infeasible experiment at this dataset scale\.

Training configuration\.AdamW optimizer, fp16 precision, no quantization, single consumer GPU \(RTX 3060, 12 GB VRAM\)\. Effective batch size 16 via gradient accumulation \(1×161\\times 16\), learning rate1×10−41\\times 10^\{\-4\}with 30 warmup steps\. LoRA dropout of 0\.05\. Audio is resampled to 16 kHz\. We train for a maximum of 300 steps over the 303\-utterance training set \(roughly 16 epochs\), evaluating on the validation set every 75 steps and selecting the final checkpoint at step 300\. Full configuration intraining/scripts/config\.pyof the accompanying repository\.

## 5\. Results

### 5\.1 Aggregate benchmark

All models are evaluated on the same 60\-utterance held\-out test split, never seen during training \(fixed seed for reproducibility\)\.

Table 2:Benchmark results on the NepFinSpeech\-403 test set \(60 held\-out utterances\)The general\-domain Nepali fine\-tune \(Dragneel/whisper\-small\-nepali, trained on the OpenSLR\-54 corpus\[[4](https://arxiv.org/html/2609.01737#bib.bib18)\]\) achieves lower WER than zero\-shot Whisper large\-v2, but critically, its NumAcc remains 0\.0%\. Inspection of its predictions reveals anumeral format mismatch: the general\-domain model outputs numerals as Nepali words \(“tin hajaar ek saya pacchis”\) rather than Devanagari digits \(3125\), since the OpenSLR training data uses word\-form transcriptions\. Even when the number is semantically correct, the downstream intent parser cannot extract a digit sequence from word\-form output\. This confirms that the improvement from our domain\-adapted model is not simply due to fine\-tuning onanyNepali data, but specifically from financial\-domain data with digit\-form transcriptions\.

The zero\-shot WER exceeds 100%, which indicates the model’s output is, on average, longer than the reference — consistent with insertion\-heavy hallucination rather than simple mistranscription\. Manual inspection of zero\-shot predictions confirms this: outputs frequently contain phonetically plausible but semantically incoherent Devanagari, including malformed numeral sequences and occasionally repeated phrase fragments\. Domain adaptation does not merely improve transcription accuracy; it recovers basic output coherence in a domain where the base model fails qualitatively, not just quantitatively\.

### 5\.2 Per\-utterance significance

Aggregate WER can be dominated by a small number of catastrophic failures\. We therefore additionally report a paired, per\-utterance comparison: for each of the 60 test utterances, we compute WER under both the zero\-shot and domain\-adapted models and compare directly\.

Table 3:Paired per\-utterance comparison \(domain\-adapted vs\. zero\-shot\)The zero\-shot model yielded a mean per\-utterance WER of 130\.33% \(95% bootstrap CI \[115\.15, 152\.96\],B=10,000B=10\{,\}000\), whereas the domain\-adapted model achieved 42\.61% \(95% bootstrap CI \[36\.76, 48\.59\]\)\.

The improvement is not driven by a handful of outliers: the domain\-adapted model is better on essentially every individual utterance in the test set, and the sign test rejects the null hypothesis of no systematic difference at an extremely high confidence level\. Given the small absolute sample size \(n=60n=60\), we report the non\-parametric sign test rather than relying on assumptions and outlier influence\.

### 5\.3 Task\-level Interpretation Accuracy

While Word Error Rate is the standard ASR benchmark, it treats all word errors equally\. In a financial voice interface, a wrong filler word is a harmless transcription error, but a mistranscribed digit is a real\-money transaction failure\. To measure practical utility, we evaluate both models using Interpretation Error Rate \(IRER\)\[[15](https://arxiv.org/html/2609.01737#bib.bib8)\], defined here as the utterance\-level Transaction Success Rate: the percentage of test utterances where the predicted transcript, when parsed by a rule\-based slot extraction pipeline, yields exactly the same intent, amount, and recipient as the ground\-truth transcript\.

Table 4:Task\-level slot extraction accuracy \(higher is better\)The domain adaptation increases the Transaction Success Rate from 1\.67% \(effectively unusable\) to 33\.33%\. While 33\.33% is still too low for a fully autonomous financial system without user confirmation steps, the relative improvement over the zero\-shot baseline is nearly 1900% — significantly larger than the 67% relative improvement seen in WER alone\. This confirms that word\-level metrics understate the task\-level impact of domain adaptation for slot\-critical applications\.

### 5\.4 Per\-intent breakdown

Table 5:WER by command intent \(heuristic classification\)The improvement holds consistently across all three groups, with the largest absolute gain on thesendcategory — both the most frequent intent in the dataset and the one with the highest numeral and proper\-noun density, consistent with the hypothesis that domain adaptation disproportionately helps exactly the structurally hardest utterances\.

### 5\.5 Acoustic Robustness

Financial voice interfaces are often used over phone lines or in noisy environments\. We synthetically degraded the test audio using theaudiomentationslibrary and re\-evaluated the domain\-adapted model\.

Table 6:WER under simulated acoustic conditionsThe model degrades gracefully under realistic mobile conditions\. GSM band\-limiting — the most relevant scenario for phone\-based payments — adds only 4\.1 percentage points of WER\. Performance collapses only at 0 dB SNR, where signal and noise are of equal power\.

### 5\.6 Data Efficiency

How much data does domain adaptation actually need? We trained our LoRA configuration on increasing subsets of the training data and evaluated each checkpoint on the full test set\.

Table 7:Data efficiency: WER and Transaction Success Rate by training set sizeWith only 50 examples, WER already drops by 43% relative to zero\-shot\. Performance improves steadily up toN=200N=200, with diminishing returns beyond that point\. The peak TSR of 38\.3% atN=300N=300and the slight regression atN=403N=403suggest that the current dataset size is near the saturation point for this LoRA rank and training configuration\. Figure[1](https://arxiv.org/html/2609.01737#S5.F1)visualizes these trends\.

Figure 1:Data efficiency learning curve\. WER \(blue, left axis\) decreases sharply with the first 100 training examples and continues to improve gradually\. Transaction Success Rate \(red, right axis\) peaks atN=300N=300\.For practitioners in other low\-resource domains, these results align with recent findings on data\-efficient LoRA adaptation\[[18](https://arxiv.org/html/2609.01737#bib.bib19)\]and indicate that a focused collection of 100–300 utterances is sufficient to achieve most of the available adaptation gain\.

## 6\. Error Analysis

### 6\.1 Numeral Confusions

While domain adaptation increased numeral exact\-match accuracy from 0\.0% to 73\.9%, we analyzed the remaining errors to understand the failure modes of the adapted model\. In 25 out of the 60 test utterances, the model transcribed exactly one numeral incorrectly\. We observed three distinct patterns in these confusions:

1. 1\.Zero Insertion/Deletion:The most common failure mode involves missing or adding trailing zeros \(e\.g\., transcribing5000as50000, or650000as65000\)\. This indicates the model occasionally struggles to distinguish the acoustic duration of repeating zero words in Nepali \(likehajaarvslaakh\)\.
2. 2\.Prefix Hallucinations:The model occasionally prepends the digit 5 to amounts \(e\.g\.,800transcribed as5800\)\. This is likely an artifact of the acoustic similarity between the Nepali word for “Rs\.” \(Rupaiya\) and the number 5 \(Paanch\), which frequently co\-occur in the training data\.
3. 3\.Similar Sounding Digits:We observed phonetic confusions between digits such as 12 and 1 \(e\.g\.,20000transcribed as120000\)\.

Figure[2](https://arxiv.org/html/2609.01737#S6.F2)shows the distribution of these error types across the 25 affected utterances\.

Figure 2:Distribution of numeral error patterns in the 25 test utterances where exactly one numeral was incorrectly transcribed\.These specific failures highlight why ASR systems for financial applications cannot rely on raw transcriptions alone and require secondary confirmation UI steps before executing transactions\.

### 6\.2 Taxonomy Gaps

Qualitative inspection of the “other” category \(Table[5](https://arxiv.org/html/2609.01737#S5.T5)\) surfaces a genuine taxonomy gap rather than annotation noise\. A recurring pattern — third\-person statements describing funds already received, e\.g\.“Archana Shrestha\-le Kumari Bank\-ma ru 4750 prapta gareka chhan”\(“Archana Shrestha received Rs\. 4750 at Kumari Bank”\) — does not map cleanly onto the send/load/balance taxonomy used elsewhere in this work, since it is neither a command nor strictly a balance query\. We did not anticipate this pattern during dataset design; it appears to have entered the corpus because some contributors interpreted “describe a financial transaction” prompts more broadly than intended\.

We treat this as a useful negative result: an intent classifier trained naively on the present three\-class scheme would silently misclassify this “receive” pattern, and any production deployment of the application’s NLP layer should add a fourth intent class or explicitly reject out\-of\-taxonomy utterances rather than forcing a best\-effort match\. We did not retrain the application’s intent parser to add this class in the current version; this is identified as future work in Section[9](https://arxiv.org/html/2609.01737#S9)\.

A second, smaller pattern in the “other” category is unrelated demographic statements \(e\.g\. stating one’s age\) appearing in place of an expected financial command\. The frequency is too low \(n=4n=4of 60 test utterances\) to support a strong claim about cause, but it is consistent with occasional prompt\-following drift during data collection rather than a model artifact, since it appears identically in both zero\-shot and domain\-adapted transcriptions of the same audio\.

## 7\. System Architecture and Deployment

The application is a Next\.js web app deployed on Vercel, communicating with a Supabase\-backed PostgreSQL database \(authentication, wallet balances, and an atomic SQL transfer function to prevent race conditions on concurrent transactions\)\. Voice commands are captured client\-side, sent to a self\-hosted FastAPI inference server \(Hugging Face Spaces, Docker runtime\) running the LoRA\-adapted model, and the resulting transcript is parsed by a two\-stage rule\-and\-confidence intent classifier before a spoken confirmation is presented to the user\. Self\-hosting the inference endpoint, rather than depending solely on a third\-party serverless inference API, was adopted after observing unreliable cold\-start behavior under the free tier of hosted inference — a practical deployment lesson as relevant as the modeling result itself for a system intended for real accessibility use\.

### 7\.1 Informal Usability Observations

Prior to the current LoRA\-adapted model and voice\-interaction redesign described in this paper, an early prototype of the application was informally tested with 3–5 sighted volunteers using blindfolds to simulate visual impairment\. No formal protocol, timing instrumentation, or written consent process was used at this stage — this was exploratory, not a controlled study, and we report it as such rather than overstating its rigor\.

Results were mixed: testers were able to complete some voice\-driven tasks, but encountered friction at several points in the interaction flow, consistent with the prototype’s reliance on the zero\-shot base ASR model \(Table[2](https://arxiv.org/html/2609.01737#S5.T2)\) and a less structured confirmation flow than the current version\. We do not have preserved logs or timing data from these sessions and do not report quantitative metrics from them\.

This informal round directly motivated three subsequent design changes: \(1\) the move to a domain\-adapted ASR model rather than the zero\-shot base, given the qualitative incoherence observed in early transcriptions \(Section[5](https://arxiv.org/html/2609.01737#S5)\); \(2\) the explicit voice\-confirmation step before any financial action is executed, added after testers expressed uncertainty about whether a command had been correctly understood; and \(3\) the self\-hosted inference deployment described above, motivated by unreliable response latency observed during testing under the previous hosted\-inference setup\.

A structured usability evaluation — ideally with visually impaired participants rather than blindfolded sighted volunteers, following a protocol such as the System Usability Scale used by Ilyasa et al\.\[[19](https://arxiv.org/html/2609.01737#bib.bib5)\]— on the current version of the system remains important future work and is the most significant remaining gap between this paper’s model\-level contribution and a complete accessibility evaluation of the deployed application\.

## 8\. Discussion

Our results show that LoRA adaptation on a small domain\-specific corpus produces disproportionately large task\-level gains relative to the word\-level improvement\. The 67\.2% relative WER reduction is itself substantial, but the Transaction Success Rate improvement from 1\.67% to 33\.33% — a roughly 20×\\timesgain — suggests that WER understates the practical impact of domain adaptation for slot\-critical applications\. This echoes observations by Kim et al\.\[[14](https://arxiv.org/html/2609.01737#bib.bib7)\]that word\-level metrics can mis\-rank ASR systems when downstream task performance is the actual objective\.

The data efficiency results have practical implications beyond Nepali financial speech\. With only 50 transcribed utterances, the adapted model already halved the zero\-shot WER, and performance plateaued around 300 examples\. For other low\-resource language–domain pairs where labeled data is expensive to collect, this suggests a realistic path: a small, focused collection effort of a few hundred utterances, combined with LoRA adaptation of a large pretrained model, may be sufficient for a usable prototype\. The slight performance drop atN=403N=403relative toN=300N=300is consistent with either mild overfitting or noise in the train/test split at this sample size\.

The 33\.33% Transaction Success Rate, while a large improvement over the 1\.67% baseline, is clearly insufficient for an autonomous payment system\. The remaining errors are concentrated in numeral transcription: the zero\-insertion/deletion pattern \(Section[6](https://arxiv.org/html/2609.01737#S6)\) accounts for the majority of transaction failures and represents a systematic weakness in how the model resolves Nepali magnitude words \(hajaar,laakh\)\. Post\-processing heuristics \(e\.g\., constraining outputs to valid numeral sequences\) or a small numeral\-specific language model could address this without retraining\.

The gap between our single\-annotator 42\.58% WER and the 10–11% CERs reported by Regmi and Bal\[[2](https://arxiv.org/html/2609.01737#bib.bib3)\]and Paudel et al\.\[[3](https://arxiv.org/html/2609.01737#bib.bib4)\]on general Nepali speech deserves careful interpretation\. Those systems were trained on orders of magnitude more data \(159k utterances\) and evaluated on read speech from the same domain\. Our WER is measured on a fundamentally harder domain \(dense numerals, proper nouns, financial jargon\) with a far smaller training set\. Direct numerical comparison across domains and evaluation protocols is not meaningful; the relevant comparison is our own zero\-shot vs\. adapted performance on the same held\-out set\.

The general\-domain baseline \(Table[2](https://arxiv.org/html/2609.01737#S5.T2)\) provides further evidence that the gain is domain\-specific, not merely a fine\-tuning artifact\. The Whisper small model fine\-tuned on 154 hours of general Nepali speech achieves 106\.32% WER on our test set — better than zero\-shot Whisper large\-v2 \(129\.95%\) but far worse than our domain\-adapted model \(42\.58%\)\. More critically, it achieves 0% numeral accuracy because the OpenSLR training data transcribes numbers in word form rather than digit form\. This numeral format mismatch renders the general\-domain model unusable for any downstream financial application, regardless of its word\-level accuracy\.

## 9\. Limitations and Future Work

Dataset scale\.403 utterances \(303 for training\) is small relative to standard ASR fine\-tuning corpora\. The results should be read as a feasibility demonstration — domain adaptation provides large relative gains even at this scale — not as a claim of state\-of\-the\-art absolute performance\. NumAcc of 73\.9% indicates meaningful remaining error on numeral transcription, the single most safety\-critical metric for a payment application; this is not yet production\-ready accuracy\.

Baseline comparison\.Our general\-domain baseline uses Whisper small \(244M parameters\) rather than Whisper large\-v2 \(1\.55B parameters\), making the comparison imperfect — the general\-domain model is disadvantaged by model size\. A controlled experiment would fine\-tune the same large\-v2 base on a size\-matched sample of general Nepali speech\. Additionally, the 0% numeral accuracy of the general\-domain baseline is partly attributable to a numeral format difference \(word\-form vs\. digit\-form transcription\) rather than purely to domain mismatch\.

Speaker overlap\.Contributor identity was not recorded during data collection\. While we verified that no exact\-duplicate transcripts straddle train/test splits, we cannot rule out that the same speaker appears in both\. Speaker\-disjoint splits would provide a more conservative performance estimate\.

Single\-annotator transcription\.Transcripts were manually verified but not independently double\-annotated; no inter\-annotator agreement statistic is available\.

Intent taxonomy gap\.As discussed in Section[6](https://arxiv.org/html/2609.01737#S6), a recognizable “funds received” utterance pattern is not represented in the current three\-class intent scheme\. Future work should add this as an explicit fourth class and re\-evaluate the application’s NLP parser\.

No formal accessibility evaluation\.No usability study has yet been conducted with visually impaired users, the system’s intended audience\. A System Usability Scale \(SUS\) study, following the methodology used by Ilyasa et al\.\[[19](https://arxiv.org/html/2609.01737#bib.bib5)\], is planned future work\.

## 10\. Conclusion

We presented NepFinSpeech\-403, a domain\-specific Nepali financial speech dataset, and showed that LoRA fine\-tuning of Whisper large\-v2 on this small corpus produces a large improvement over the zero\-shot baseline: a 67\.2% relative WER reduction, recovery of basic output coherence in a domain where the base model produces incoherent transcriptions, and improvement on 59 of 60 individual test utterances \(p=3\.5×10−18p=3\.5\\times 10^\{\-18\}\)\. The gain is consistent across command types\. Error analysis identified both systematic numeral confusion patterns \(zero insertion/deletion, prefix hallucination\) and a genuine gap in our intent taxonomy\. A data efficiency sweep showed that as few as 100 utterances are sufficient to halve the zero\-shot WER, and that performance plateaus around 300 examples — a finding directly relevant to practitioners building domain\-adapted ASR for other low\-resource languages\. The system is deployed as a working voice\-first eWallet application, demonstrating that the distance between a research result and a usable accessibility tool is bridgeable with modest additional engineering\.

## Ethics Statement

All audio contributors voluntarily recorded prompted phrases through a web platform and consented to release of their recordings under CC\-BY 4\.0\. No personally identifiable information \(names, phone numbers, or account details\) was collected or stored\. The dataset contains only scripted financial commands, not real transactions\. The deployed application operates on simulated wallet balances and does not process real money\. Contributor identity was not tracked per utterance, which limits our ability to report demographic statistics but also minimizes privacy risk\.

## Data and Code Availability

## References

- \[1\]\(2019\)World report on vision\.External Links:[Link](https://www.who.int/publications/i/item/9789241516570)Cited by:[§1](https://arxiv.org/html/2609.01737#S1.p1.1)\.
- \[2\]S\. Regmi and B\. K\. Bal\(2021\)An end\-to\-end speech recognition for the nepali language\.InProceedings of the 18th International Conference on Natural Language Processing,Cited by:[§1](https://arxiv.org/html/2609.01737#S1.p2.1),[§2](https://arxiv.org/html/2609.01737#S2.p1.1),[§8](https://arxiv.org/html/2609.01737#S8.p4.1)\.
- \[3\]S\. Paudel, B\. K\. Bal, and D\. Shrestha\(2023\)Large vocabulary continuous speech recognition for Nepali language using CNN and transformer\.InProceedings of LDK,Cited by:[§1](https://arxiv.org/html/2609.01737#S1.p2.1),[§2](https://arxiv.org/html/2609.01737#S2.p1.1),[§8](https://arxiv.org/html/2609.01737#S8.p4.1)\.
- \[4\]O\. Kjartansson, S\. Sarin, K\. Pramodh, R\. Dua,et al\.\(2018\)Large vocabulary speech recognition for languages of Nepal\.Note:OpenSLR\-54 Nepali speech corpusExternal Links:[Link](https://www.openslr.org/54/)Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p1.1),[§5\.1](https://arxiv.org/html/2609.01737#S5.SS1.p2.1)\.
- \[5\]A\. Baevski, Y\. Zhou, A\. Mohamed, and M\. Auli\(2020\)Wav2vec 2\.0: a framework for self\-supervised learning of speech representations\.InProceedings of NeurIPS,Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p2.1)\.
- \[6\]A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever\(2023\)Robust speech recognition via large\-scale weak supervision\.InProceedings of ICML,Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p2.1),[§4](https://arxiv.org/html/2609.01737#S4.p1.1)\.
- \[7\]A\. Conneau, M\. Ma, S\. Khanuja, Y\. Zhang, V\. Axelrod, S\. Dalmia, J\. Riber, A\. Bapna,et al\.\(2023\)FLEURS: few\-shot learning evaluation of universal representations of speech\.InProceedings of SLT,Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p2.1)\.
- \[8\]V\. Pratap, A\. Tjandra, B\. Shi, P\. Tober, A\. Babu, S\. Kunber, A\. Graves, and M\. Auli\(2024\)Scaling speech technology to 1000\+ languages\.Journal of Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p2.1)\.
- \[9\]T\. Javed, S\. Doddapaneni, A\. Raman, K\. Bali, M\. M\. Khapra,et al\.\(2023\)IndicSUPERB: a speech processing universal performance benchmark for Indian languages\.InProceedings of ACL,Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p2.1)\.
- \[10\]S\. Gandhi, P\. von Platen, and A\. Rush\(2024\)Distil\-whisper: robust knowledge distillation via large\-scale pseudo labelling\.InProceedings of ACL,Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p2.1)\.
- \[11\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InProceedings of ICLR,Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p3.1),[§4](https://arxiv.org/html/2609.01737#S4.p2.1)\.
- \[12\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)QLoRA: efficient finetuning of quantized language models\.InProceedings of NeurIPS,Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p3.1)\.
- \[13\]Y\. Chenet al\.\(2024\)Adapting Whisper for domain\-specific ASR: a comparison of full fine\-tuning, LoRA, and prompt tuning\.arXiv preprint arXiv:2407\.11756\.Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p3.1)\.
- \[14\]S\. Kim, A\. Arora, D\. Le, C\. Yeh, C\. Fuegen, O\. Kalinli, and M\. L\. Seltzer\(2021\)Semantic distance: a new metric for ASR performance analysis towards spoken language understanding\.InProc\. Interspeech 2021,Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p4.1),[§8](https://arxiv.org/html/2609.01737#S8.p1.1)\.
- \[15\]X\. Fu, F\. Chang, M\. Radfar, K\. Wei, J\. Liu, G\. P\. Strimel, and K\. M\. Sathyendra\(2022\)Multi\-task RNN\-T with semantic decoder for streamable spoken language understanding\.External Links:2204\.00558Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p4.1),[§5\.3](https://arxiv.org/html/2609.01737#S5.SS3.p1.1)\.
- \[16\]D\. Ogucheet al\.\(2026\)Toward accessible mobile money: a voice\-driven, biometrically secured USSD automation framework for visually impaired users\.External Links:2605\.31375Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p5.1)\.
- \[17\]R\. Sharmaet al\.\(2026\)A voice biometric UPI transaction system for visually impaired users\.International Journal of Scientific Development and Research \(IJSDR\)\.Cited by:[§2](https://arxiv.org/html/2609.01737#S2.p5.1)\.
- \[18\]Y\. Shinoharaet al\.\(2024\)Data\-efficient domain adaptation for ASR with lora: how little data do you need?\.arXiv preprint arXiv:2404\.14043\.Cited by:[§5\.6](https://arxiv.org/html/2609.01737#S5.SS6.p3.1)\.
- \[19\]I\. Ilyasa, R\. Fauzi, and S\. Suakanto\(2023\)Creating a mobile E\-Wallet for visual impairment using an inclusive design approach\.InProceedings of ICOEINS,Cited by:[§7\.1](https://arxiv.org/html/2609.01737#S7.SS1.p4.1),[§9](https://arxiv.org/html/2609.01737#S9.p6.1)\.