Efficiently Adapting Spoken Language Models for the Singaporean Context

# Efficiently Adapting Spoken Language Models for the Singaporean Context
Source: [https://arxiv.org/html/2607.10092](https://arxiv.org/html/2607.10092)
\(July 10, 2026\)

###### Abstract

Spoken language models \(SLMs\) unify speech perception and reasoning, but adapting them to sensitive domains is underexplored, especially when the original training data is inaccessible and the use case demands multilingual, spoken\-query interaction\. We adapt an open\-source SLM to the Singaporean Home Team context across five speech tasks in Singapore’s four official languages, combining LoRA fine\-tuning, a surrogate text\-QA dataset that guards against catastrophic forgetting, and a multi\-task objective that adapts the CoBa reweighting scheme to speech\. We also build HTD\-multilingual\-QA, a 504,853 sample multilingual QA dataset in text and spoken form\. The resulting HT\-Moonstone \(5B\) matches or outperforms SLMs up to 7× its size on most tasks, attains the best accent and gender recognition among all models evaluated, and loses under 2% of its original speech QA ability\.

Efficiently Adapting Spoken Language Models for the Singaporean Context

Language AI R&D, xDataHome Team Science & Technology Agency \(HTX\), SingaporeNg Jia Sheng Jason

## 1Introduction

The Home Team Science and Technology Agency \(HTX\) is a statutory board under Singapore’s Ministry of Home Affairs \(MHA\)\. It serves as a “force multiplier” for the Home Team departments, applying science and technology to strengthen homeland security and public safety\. Among the use cases HTX receives from these departments are speech tasks such as automatic speech recognition \(ASR\) and spoken question answering \(QA\), the latter forming the core of speech\-to\-speech conversational AI applications\. Because much of this work involves highly sensitive data, closed\-source or proprietary models are often unsuitable, which is why we turn to open\-source alternatives\.

HTX’s Language AI R&D team has been developing and adapting open\-source foundational speech models, such as those for ASR\. However, the field is moving towards spoken language models \(SLMs\)\(Aroraet al\.,[2025](https://arxiv.org/html/2607.10092#bib.bib1)\), which handle a wide range of cognitive and perception speech tasks\(Penget al\.,[2024](https://arxiv.org/html/2607.10092#bib.bib2)\)\. Several open\-source SLMs already exist, including Kimi\-Audio\(Kimi Team,[2025](https://arxiv.org/html/2607.10092#bib.bib3)\)and Audio\-Flamingo\-Next\(Ghoshet al\.,[2026](https://arxiv.org/html/2607.10092#bib.bib4)\), as well as models built for the Singaporean context, such as MERaLiON\-2\(MERaLiON Team,[2024](https://arxiv.org/html/2607.10092#bib.bib5)\)\. However, because our operating environment and use cases are specific to the Home Team context, a suitable model must either be built from scratch or adapted from an existing open\-source SLM\. Training from scratch demands far more audio data, which is costly to acquire, so we favour adaptation\. The SLM domain is still nascent, and few studies document how to adapt these models effectively for new tasks while preserving their original performance, particularly when the full dataset used to build them is unavailable\.

As we have use cases involving the use of text context and spoken queries and no Singaporean multilingual spoken QA training dataset currently exists, the development of a resource that covers Singaporean English, Mandarin, Bahasa Melayu, and Tamil is essential for adapting models to handle such spoken QA task effectively\. Furthermore, adapting models without full access to their original training data often risks catastrophic forgetting\. Our adaptation strategy is designed to mitigate this risk, ensuring we maximise performance on both perceptive and cognitive speech tasks in the Singaporean domain while preserving the model’s foundational capabilities\.

In this paper, we address these challenges and introduce:

1. 1\.a strategy to effectively adapt SLMs to the Singaporean context for multiple speech tasks;
2. 2\.HTD\-multilingual\-QA: a multilingual, multi\-turn QA training dataset in both text and spoken form, grounded in Singaporean Home Team context that covers the 4 Singaporean languages \- English, Mandarin, Bahasa Melayu, and Tamil for training SLMs on spoken QA tasks; and
3. 3\.HT\-Moonstone, an adapted 5B\-parameter SLM that performs competitively for both perceptive and cognitive speech tasks in the Singaporean domain against open\-source models up to 7×\\timesits size while exhibiting only a 2% degradation in its original speech QA capabilities\.

![Refer to caption](https://arxiv.org/html/2607.10092v1/images/ht_moonstone_graphic.png)Figure 1:Examples of target tasks that can be handled by HT\-Moonstone\. Inputs for Multilingual Spoken QA are different \(text context, audio query\) compared to the rest of the tasks \(audio context, text query\)\.
## 2Related Work

### 2\.1Rise of Spoken Language Models

The paradigm of speech processing is transitioning from specialised, single\-task architectures to versatile, multi\-task capable SLMs\. Commonly known approaches typically unify audio encoders such as Whisper\(Radfordet al\.,[2022](https://arxiv.org/html/2607.10092#bib.bib17)\)with pre\-trained Large Language Models \(LLMs\) as decoders via an intermediate adapter layer\. This architecture maps audio embeddings into a latent space compatible with textual tokenisation, enabling direct audio\-to\-text processing\. By replacing conventional cascaded pipelines, which decouple ASR from downstream reasoning, these models not only reduce latency but also mitigate the”broken telephone”effect, which is the propagation of transcription errors that inherently undermines performance in downstream speech QA tasks\. Processing audio representations directly also preserves paralinguistic information beyond lexical content, enabling the decoder to reason over cues such as prosody and affect for tasks like emotion recognition\. Examples of SLMs include Voxtral\(Mistral AI,[2025](https://arxiv.org/html/2607.10092#bib.bib6)\), Audio Flamingo\(Ghoshet al\.,[2026](https://arxiv.org/html/2607.10092#bib.bib4)\)and MERaLiON\-2\(MERaLiON Team,[2024](https://arxiv.org/html/2607.10092#bib.bib5)\), which has been developed specifically to address the nuances of the Singaporean context\.

Despite the emergence of these models, adapting them to the needs of specific, sensitive environments such as the Singaporean Home Team remains an open challenge\. Existing adaptation literature are often limited to single task adjustments such as for ASR\. Furthermore, existing SLM frameworks are frequently developed for scenarios involving text\-based queries, leaving the more complex paradigm of text\-context and spoken\-query interaction relatively underexplored\.

### 2\.2Singaporean Audio\-Text Datasets

The linguistic landscape of Singapore is defined by a quadrilingual standard of English, Mandarin, Bahasa Melayu, and Tamil, which is often characterised by unique grammatical structures and code\-mixing\. These features are exemplified bySinglish, where localized syntax and lexicon are frequently interspersed with vocabulary from regional languages\. For instance, the query ”Eh, you makan already a not ah?” \(which means ”Hey, have you eaten?”\) illustrates the seamless integration of the Bahasa Melayu termmakan, meaning eat, into a colloquial English syntactic frame complete with particle\-driven modality\. Such phenomena, where conversational intent is conveyed through a fluid combination of tone, particles, and multilingual lexicon, create a significant challenge for generic spoken language models trained on monolithic, standardised datasets\. The specific prosody and conversational dynamics inherent to the unique Singaporean speech necessitate the development of localised datasets to effectively adapt open\-source models to the Singaporean domain\. While existing resources like the Multitask\-National\-Speech\-Corpus \(MNSC\)\(Wanget al\.,[2025b](https://arxiv.org/html/2607.10092#bib.bib8)\)have provided a foundation for tasks like ASR and speech QA, defined therein as audio\-based contexts paired with textual queries, they do not fully encapsulate the complexity of spontaneous, multilingual conversational speech\. Crucially, our operational use cases to use SLMs for speech\-2\-speech conversational AI require a paradigm shift because the model must process textual context to ensure factual grounding while responding to spoken queries across all four Singaporean languages\. While SQuAD\-SRC\(Tang and Tung,[2023](https://arxiv.org/html/2607.10092#bib.bib18)\)encompasses the idea of spoken queries, it does not cover the spoken style of Singaporean speech nor the other 3 Mother Tongue languages spoken in Singapore\. This establishes a clear research opportunity to develop such resources\.

### 2\.3Adapting SLMs

Two broad strategies exist to adapt SLMs for a particular domain, language and/or tasks\. The first is to assemble pre\-trained encoder and decoders and align the whole architecture over large\-scale corpora; this is the route taken by the Singapore\-oriented MERaLiON\-2\(MERaLiON Team,[2024](https://arxiv.org/html/2607.10092#bib.bib5)\)model, as well as by Southeast Asian systems such as Typhoon\-Audio\(Manakulet al\.,[2024](https://arxiv.org/html/2607.10092#bib.bib9)\)and SeaLLMs\-Audio\(Liuet al\.,[2025](https://arxiv.org/html/2607.10092#bib.bib10)\)\. While effective, this strategy is expensive and presupposes access to hundreds of thousands of audio hours of training data with huge computing power, making it ill\-suited to data and resource\-constrained conditions\.

The second strategy, and the one we pursue, is the post\-pretraining adaptation of an existing, pretrained SLM through parameter\-efficient fine\-tuning \(PEFT\)\. This route is considerably more economical, yet it remains comparatively underexplored\. CantoASR\(Chenet al\.,[2025](https://arxiv.org/html/2607.10092#bib.bib11)\)instruction\-tunes Qwen2\-Audio with LoRA\(Huet al\.,[2021](https://arxiv.org/html/2607.10092#bib.bib14)\)for low\-resource Cantonese tone and accent correction, while\(Choiet al\.,[2025](https://arxiv.org/html/2607.10092#bib.bib12)\)cross\-lingually adapts Qwen2\-Audio to low\-resource languages from minimal target\-language speech\. Most relevant to the Singaporean context,\(Quanget al\.,[2026](https://arxiv.org/html/2607.10092#bib.bib13)\)applies direct preference optimisation to MERaLiON\-2, Phi\-4\-multimodal, and Qwen2\-Audio to improve English–Mandarin code\-switched recognition on Singapore–Malaysia conversational speech\. As foreshadowed in Section[2\.1](https://arxiv.org/html/2607.10092#S2.SS1), however, these efforts share two limitations: they overwhelmingly optimise a single transcription objective \(ASR\), and they target the standard text\-query setting rather than the spoken\-query, text\-response interaction central to one of our use cases\. To our knowledge, no prior work adapts a pretrained SLM to both perception and cognitive speech tasks via a multi\-task training approach, especially for the spoken\-query setting\. Our work addresses this gap, demonstrating that an existing SLM can be efficiently adapted to the Singaporean domain via LoRA\(Huet al\.,[2021](https://arxiv.org/html/2607.10092#bib.bib14)\), without the cost of constructing a Singaporean model from scratch\.

## 3SLM Adaptation Methodology

### 3\.1Base Model Selection

Table 1:Zero\-shot benchmark of candidate base SLMs across the five target tasks, arranged in ascending order of model size\. Details of datasets used for each tasks are listed in Appendix[A](https://arxiv.org/html/2607.10092#A1)\. Best result per column inbold\. \(↑\\uparrow/↓\\downarrowindicate that higher/lower is better\.\)Prior to any fine\-tuning, we first establish which pretrained SLM offers the strongest foundation for adaptation to the Singaporean domain\. To this end, we benchmarked a set of candidate open\-source models across five perception and cognitive speech tasks chosen to reflect our target use cases:

1. 1\.Automatic Speech Recognition \(ASR\) \- EN:transcription of Singaporean\-accented English\.
2. 2\.Accent Recognition \(AR\):identification of ethnic groups among Singaporean speakers\.
3. 3\.Gender Recognition \(GR\):identification of Singaporean speakers’ gender\.
4. 4\.Spoken Question Answering \(Spoken QA\):generation of a textual answer, given a spoken question in one of the four official languages\.
5. 5\.Speech Question Answering \(Speech QA\):generation of a textual answer, given a Singaporean context in spoken form and a textual question in English\.

The evaluation sets for ASR, AR, and GR are drawn from the MNSC dataset \(ASR\-PART1\-Test, PQA\-AR\-Dialogue\-Test, and PQA\-GR\-Dialogue\-Test, respectively\)\. The Speech QA set comprises of textual questions using public Singaporean speech as context\(Wanget al\.,[2025a](https://arxiv.org/html/2607.10092#bib.bib19)\)while the Spoken QA set was developed in\-house, as detailed in Section[3\.2\.1](https://arxiv.org/html/2607.10092#S3.SS2.SSS1)\. All models were evaluated using an adapted version of the AudioBench\(Wanget al\.,[2025a](https://arxiv.org/html/2607.10092#bib.bib19)\)evaluation framework with a GPT\-4o judge and the default evaluation prompts\. Table[1](https://arxiv.org/html/2607.10092#S3.T1)reports each candidate’s results\. No single model dominated across all five tasks\. We observed some outlier results; on investigation, these arose because the model either declined to answer or produced responses that were insufficiently specific\. We selectedVoxtral\-Mini\-3B\-2507as our base model on the basis of three considerations: \(1\) its competitively low WER for Singaporean English ASR task, \(2\) its comparable Speech QA accuracy given its small size, and \(3\) its capacity to process up to 40 minutes of audio in a single pass\(Mistral AI,[2025](https://arxiv.org/html/2607.10092#bib.bib6)\), which is essential for our long\-form audio analysis use cases\. While its accent and gender recognition accuracy is comparatively low, we prioritised speech QA performance, which more closely reflects our intended use cases\.

### 3\.2Training Datasets

Table 2:Maximum available training samples, grouped by task domain\. ASR is further split into language sub\-domains\.Task DomainSub\-domainDatasetMax\. SamplesASREnglish \[EN\]MNSC ASR \(PART1,2,3,5,6\)4,956,791Mandarin \[ZH\]AISHELL\-1120,098SEAME \(ZH\-CS\)95,853AISHELL\-356,936FLEURS \[ZH\]3,246Malay \[MS\]Mesolitica \(MS\-CS\)17,851FLEURS \[MS\]2,667Tamil \[TA\]SLR12769,583SLR653,427FLEURS \[TA\]2,367Accent Recognition \(AR\)MNSC PQA\-AR4,862,485Gender Recognition \(GR\)MNSC PQA\-GR4,862,485Text Instruction FollowingNemotron\-SFT\-IF\-Chat\-v2 \(reasoning\_off\)†1,063,555Conversational QAHTD Multilingual QA \(Text\)504,853HTD Multilingual QA \(Spoken\)504,853Total17,127,050

†Estimated from average line size; exact count not precomputed\.

Guided by the tasks of interest established in Section[3\.1](https://arxiv.org/html/2607.10092#S3.SS1), we curated the training datasets listed in Table[2](https://arxiv.org/html/2607.10092#S3.T2)to adapt the base SLM\. The table reports the maximum number of samples available in each dataset, categorised by their dataset domains; the actual training mix is sub\-sampled according to a fixed proportion rather than using all samples\. The development of the HTD\-Multilingual\-QA datasets \(in both text and spoken form\) is described in Section[3\.2\.1](https://arxiv.org/html/2607.10092#S3.SS2.SSS1), and our motivation for including the Nemotron\-SFT\-Instruction\-Following\-Chat\-v2 dataset is discussed in Section[3\.2\.2](https://arxiv.org/html/2607.10092#S3.SS2.SSS2)\.

#### 3\.2\.1Development of HTD\-multilingual\-QA Dataset

To develop conversational models capable of understanding Singaporean\-style inputs, we previously built a Singaporean multi\-turn conversational QA dataset grounded in Home Team context111[https://medium\.com/htx\-dsai/how\-we\-talk\-generating\-singaporean\-conversations\-54aea9b6892b](https://medium.com/htx-dsai/how-we-talk-generating-singaporean-conversations-54aea9b6892b)and based it on a range of Singaporean personas\. Building on this dataset, we augmented it by translating equal proportions of the user turns into the other Singaporean languages \(code\-mixed with English\) using GPT\-4o, yielding the textual variant of the HTD\-Multilingual\-QA dataset\. The user turns were then converted to spoken form with OmniVoice\(Zhuet al\.,[2026](https://arxiv.org/html/2607.10092#bib.bib15)\), using voice cloning seeded from two manually recorded audio clips of Singaporean speakers per language\. The recorded voices stay true to the speakers’ ethnicities; for example, speakers of Chinese ethnicity were recorded speaking code\-switched Mandarin\. During spoken\-form generation, we found that roughly 0\.2% of the translated user turns were erroneous, exhibiting repeated tokens or altered meaning; these were corrected afterwards\. Assistant turns were not converted to spoken form\. In total, this yields 504,853 samples comprising 1,723,693 QA pairs distributed across 2 to 8 turns, in each of the text and spoken variants\.

#### 3\.2\.2Usage of Surrogate Datasets

Since we intend to adapt the SLM chosen in Section[3\.1](https://arxiv.org/html/2607.10092#S3.SS1)to other speech tasks in the Singaporean domain, we mitigate catastrophic forgetting through PEFT\. Beyond this, we specifically aim to preserve the model’s original instruction\-following and QA capabilities\. Because the original training data is unavailable to us, we instead seek to identify a surrogate dataset capable of maintaining the model’s speech QA performance during adaptation\. To this end, we curated a set of both text QA and speech QA datasets and evaluated their effectiveness in preserving spoken QA and speech QA capabilities in the English language\. For our benchmarks, we focus on both speech QA and spoken QA rather than text QA capabilities because this is what our downstream use cases require\.

Table 3:Text/Speech QA candidate surrogate dataset benchmarks\. Details of datasets used for each tasks are listed in Appendix[A](https://arxiv.org/html/2607.10092#A1)\.Across our PEFT experiments on each candidate dataset listed in Table[3](https://arxiv.org/html/2607.10092#S3.T3), in which the SLM was fine\-tuned on 20k samples each, Nemotron\-SFT\-Instruction\-Following\-Chat\-v2 emerged as the most effective at mitigating catastrophic forgetting\. It was the only dataset to improve performance on both tasks; the remaining datasets degraded one or both scores\. Notably, it achieved these gains despite being a text\-only QA dataset, whereas the evaluation operates directly on speech\-modality inputs\.

### 3\.3Multi\-Task Training

Adapting the SLM to the Singaporean context introduces additional complexity when several task domains must be learned simultaneously, which leads to two identified issues\. Firstly, the per\-batch loss cannot simply be aggregated over all tokens; deeper cognitive speech task domains such as QA emit far more tokens than shallow cognitive task domains such as gender recognition, so token\-level averaging systematically suppresses the contribution of the latter\. Secondly, the evaluation loss of each task domain converges at a different point during training, so tasks that converge early are prone to overfitting while later\-converging tasks are still improving\.

To address the first issue, we decouple a sample’s loss from its answer length by averaging the cross\-entropy over each sample’s own answer tokens before aggregating across samples,

ℓ​\(x\)\\displaystyle\\ell\(x\)=1\|y\|​∑j=1\|y\|CE​\(yj,y^j\),\\displaystyle=\\frac\{1\}\{\|y\|\}\\sum\_\{j=1\}^\{\|y\|\}\\mathrm\{CE\}\\\!\\left\(y\_\{j\},\\hat\{y\}\_\{j\}\\right\),\(1\)ℒi\\displaystyle\\mathcal\{L\}\_\{i\}=1\|Bi\|​∑x∈Biℓ​\(x\)\.\\displaystyle=\\frac\{1\}\{\|B\_\{i\}\|\}\\sum\_\{x\\in B\_\{i\}\}\\ell\(x\)\.
whereyyis the target answer of samplexx,BiB\_\{i\}is the set of samples in the batch belonging to taskii, andℒi\\mathcal\{L\}\_\{i\}is the resulting per\-task\-domain loss\. The per\-task\-domain loss is aggregated through the use of weights that can be adjusted throughout the training process,

ℒ​\(t\)=∑i=1Kωi​\(t\)​ℒi\.\\mathcal\{L\}\(t\)=\\sum\_\{i=1\}^\{K\}\\omega\_\{i\}\(t\)\\mathcal\{L\}\_\{i\}\.\(2\)
![Refer to caption](https://arxiv.org/html/2607.10092v1/x1.png)Figure 2:Per dataset domain evaluation loss over training for the four 256k\-sample runs, under equal versus redistributed dataset proportion conditions and with or without CoBa\.The second issue concerns task domains converging at different points, which leaves early\-converging task domains prone to overfitting while later ones are still improving\. This motivates adjustingωi​\(t\)\\omega\_\{i\}\(t\)over the course of training rather than holding it fixed, and we test the idea through an ablation on dynamic weight adaptation\. The method we evaluate, CoBa\(Gonget al\.,[2024](https://arxiv.org/html/2607.10092#bib.bib16)\), sets each task domain’s weight from its absolute and relative evaluation loss, down\-weighting early\-converging task domains and up\-weighting late\-converging ones, which is precisely the behavior the second issue calls for\. Because the original CoBa experiments were confined to text\-modality tasks, we ran these ablations also to test whether the method transfers to the speech\-modality tasks of interest\. We evaluated it under two task\-domain\-proportion conditions: an equal\-proportion condition, and a redistributed condition with less GR and AR and more Text\-IF and Conversational QA domain datasets\. The redistributed condition was motivated by our observation that the evaluation losses of accent and gender recognition converge early, freeing budget that can be reallocated to Text\-IF and Conversational QA task domains that have not yet converged\. This gives four runs in total \(CoBa and non\-CoBa under each condition\), all using 256k training samples under identical training parameters\. The results are shown in Figure[2](https://arxiv.org/html/2607.10092#S3.F2)\. For AR and GR, the final evaluation loss is comparable across all four runs\. For ASR, both non\-CoBa runs achieve lower loss than their CoBa counterparts under either condition\. For Text\-IF and Conversational QA, CoBa under the redistributed condition attains the lowest loss\. Since Conversational QA performance is our use\-case priority, we adopted the CoBa redistributed strategy for its lowest Conversational QA evaluation loss, accepting the trade\-off in ASR\.

### 3\.4Results

Adopting the CoBa redistributed strategy mentioned in Section[3\.3](https://arxiv.org/html/2607.10092#S3.SS3), we scaled the training set up to 1 million samples, amounting to an estimated 3,000 hours of audio data and 500M text tokens used to adapt the Voxtral\-Mini\-3B\-2507 model\. Training was carried out on 2×\\timesNVIDIA H100 NVL GPUs \(94 GB each\) with a per\-device batch size of 1, gradient accumulation of 8, a single epoch over 62,500 steps, a learning rate of1×10−51\\times 10^\{\-5\}, and LoRA hyperparametersr=32r=32,α=64\\alpha=64, and dropout=0\.05=0\.05\. Training completed in 100 hours, yielding our adapted model for the Singaporean context,HT\-Moonstone\.

Table 4:Benchmark of candidate SLMs versus HT\-Moonstone across the five target tasks\. Details of datasets used for each tasks are listed in Appendix[A](https://arxiv.org/html/2607.10092#A1)\. Best result per column inbold\. \(↑\\uparrow/↓\\downarrowindicate that higher/lower is better\.\)Table 5:Benchmark of candidate SLMs versus HT\-Moonstone for multilingual ASR\. Details of datasets used for each tasks are listed in Appendix[A](https://arxiv.org/html/2607.10092#A1)\. Best result per column inbold\. \(lower is better\.\)Table[4](https://arxiv.org/html/2607.10092#S3.T4)compares HT\-Moonstone against the benchmarked models across the target speech tasks\. Most strikingly, HT\-Moonstone improves over its base model, Voxtral\-Mini\-3B\-2507, on four of the five tasks: raising accent recognition from 24\.7 to 72\.6 and gender recognition from 15\.7 to 93\.9, while sacrificing only roughly one accuracy point on Speech QA, demonstrating effective adaptation while minimising catastrophic forgetting\. On accent and gender recognition it attains the best result of any model, surpassing systems up to 7×\\timesits size\. On Spoken QA it is second only to Qwen3\-Omni, a model 7×\\timeslarger, while outperforming all others including its base 24B Voxtral\-Small variant\. On Speech QA it remains within 5 accuracy points of the best\-performing model\. We further evaluate HT\-Moonstone against SLMs trained from scratch for multilingual ASR across the four languages of Singapore, including Mandarin and Bahasa Melayu code\-switching; as shown in Table[5](https://arxiv.org/html/2607.10092#S3.T5), HT\-Moonstone reduces its base model’s average error rate from 52\.4 to 14\.6, trailing the purpose\-built MERaLiON\-2\-10B by only one error rate point, and achieving the best result of all on Malay code\-switched speech\.

## 4Limitations

We have identified the following limitations in our adaptation methodology:

- •Model size:Owing to compute constraints, our experiments were confined to a single 5B\-parameter SLM \(Voxtral\-Mini\-3B\-2507\)\. We were therefore unable to verify whether our findings on catastrophic forgetting, task\-aggregated loss, and CoBa\-based weighting hold at larger scales, where convergence dynamics and the capacity for multi\-task learning may differ\.
- •Task coverage:Our study focuses on a fixed set of speech tasks\. We shortlisted additional candidates, such as audio\-scene QA and audio emotion recognition, as promising directions for extending the model’s capabilities, but were unable to investigate them within the time available\. A broader and more diverse task suite would give a more complete picture of how well the approach generalises across speech tasks\.
- •Model architecture:We did not explore swapping out architectural components of the SLM, and whether alternative SLM architectural designs yield better multi\-task adaptation remains an open question we hope to pursue\.
- •Hyperparameter search:Our exploration of the training data mixture was necessarily limited\. A more exhaustive search over task domain proportions rather than the heuristic reallocation adopted in the redistributed conditions mentioned in Section[3\.3](https://arxiv.org/html/2607.10092#S3.SS3)may yield a more optimal mixture and further improve the resultant model\.
- •Data imbalance across languages:As shown in Table[2](https://arxiv.org/html/2607.10092#S3.T2), the Tamil and Malay ASR datasets are substantially smaller than those for the other languages\. This imbalance prevented us from scaling up the total training set further without over\-sampling these low\-resource languages\. Acquiring additional Tamil and Malay ASR data could further reduce evaluation losses for ASR, Text\-IF and Conversational QA task domains and improve general QA performance\.
- •Scope of evaluation:Our evaluation focuses on speech\-centric tasks\. We did not conduct a comprehensive assessment on broader text\-only benchmarks, leaving the extent to which the model’s general text\-modality capabilities are preserved an open question\.

## 5Conclusion

We presented a strategy for adapting an open\-source spoken language model to the Singaporean context without access to its original training data\. By combining LoRA\-based parameter\-efficient fine\-tuning, a surrogate text\-QA grounding dataset to guard against catastrophic forgetting, and a multi\-task objective that averages cross\-entropy per task domain and adapts the CoBa convergence balancer to speech tasks, we adapted Voxtral\-Mini\-3B\-2507 across five perception and cognitive speech tasks spanning Singapore’s four official languages\. To enable the spoken\-query, text\-response setting central to our use cases, we constructedHTD\-multilingual\-QA, a multilingual QA dataset in both text and spoken form grounded in Singaporean Home Team context\. The resulting model,HT\-Moonstone, matches or outperforms open\-source SLMs up to 7×\\timesits size on most target tasks, attaining the best accent and gender recognition accuracy of any model evaluated, while incurring under 2% degradation in its original speech QA capability\. These results show that targeted, economical adaptation is a viable alternative to training from scratch for specialised, data\- and resource\-constrained domains\. Future work includes validating the approach at larger model scales, broadening the task suite and evaluation benchmarks, and exploring alternative architectural components such as the audio encoder\.

## Acknowledgments

The author would like to thank the management and colleagues at Home Team Science & Technology Agency \(HTX\) especially Chow Yew Wah, Lim Ming En, Choy Xin Yun Calista, Lye En Lih, Chan Shing Yee and our director Lim Kian Boon, for their support, valuable feedback, and the resources provided throughout this work\. Every reasonable effort has been made to acknowledge and cite the original sources of information, ideas, figures, and other materials used in this work\. Any inadvertent omissions are unintentional, and the author welcome corrections where appropriate\.

## References

- S\. Arora, K\. Chang, C\. Chien, Y\. Peng, H\. Wu, Y\. Adi, E\. Dupoux, H\. Lee, K\. Livescu, and S\. Watanabe \(2025\)On the landscape of spoken language models: a comprehensive survey\.arXiv preprint arXiv:2504\.08528\.Cited by:[§1](https://arxiv.org/html/2607.10092#S1.p2.1)\.
- D\. Chen, Y\. Lin, Y\. Huang, Z\. Gong, D\. Jiang, Z\. Xie, and Y\. R\. Fung \(2025\)CantoASR: prosody\-aware ASR\-LALM collaboration for low\-resource Cantonese\.arXiv preprint arXiv:2511\.04139\.Cited by:[§2\.3](https://arxiv.org/html/2607.10092#S2.SS3.p2.1)\.
- Y\. Choi, J\. Jung, H\. Kim, H\. Nguyen, and H\. Kim \(2025\)Exploring fine\-tuning of large audio language models for spoken language understanding under limited speech data\.arXiv preprint arXiv:2509\.15389\.Cited by:[§2\.3](https://arxiv.org/html/2607.10092#S2.SS3.p2.1)\.
- S\. Ghosh, A\. Goel, K\. Jayakumar, L\. Koroshinadze, N\. Anand, Z\. Kong, S\. Gururani, S\. Lee, J\. Kim, A\. Aljafari, C\. H\. Yang, S\. Kim, R\. Duraiswami, D\. Manocha, M\. Shoeybi, B\. Catanzaro, M\. Liu, and W\. Ping \(2026\)Audio Flamingo Next: next\-generation open Audio\-language models for speech, sound, and music\.arXiv preprint arXiv:2604\.10905\.Cited by:[§1](https://arxiv.org/html/2607.10092#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.10092#S2.SS1.p1.1)\.
- Z\. Gong, H\. Yu, C\. Liao, B\. Liu, C\. Chen, and J\. Li \(2024\)CoBa: convergence balancer for multitask finetuning of large language models\.arXiv preprint arXiv:2410\.06741\.Cited by:[§3\.3](https://arxiv.org/html/2607.10092#S3.SS3.p6.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2021\)LoRA: low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685\.Cited by:[§2\.3](https://arxiv.org/html/2607.10092#S2.SS3.p2.1)\.
- Kimi Team \(2025\)Kimi\-Audio technical report\.arXiv preprint arXiv:2504\.18425\.Cited by:[§1](https://arxiv.org/html/2607.10092#S1.p2.1)\.
- C\. Liu, M\. Aljunied, G\. Chen, H\. P\. Chan, W\. Xu, Y\. Rong, and W\. Zhang \(2025\)SeaLLMs\-Audio: large audio\-language models for Southeast Asia\.arXiv preprint arXiv:2511\.01670\.Cited by:[§2\.3](https://arxiv.org/html/2607.10092#S2.SS3.p1.1)\.
- P\. Manakul, G\. Sun, W\. Sirichotedumrong, K\. Tharnpipitchai, and K\. Pipatanakul \(2024\)Enhancing low\-resource language and instruction following capabilities of audio language models\.arXiv preprint arXiv:2409\.10999\.Cited by:[§2\.3](https://arxiv.org/html/2607.10092#S2.SS3.p1.1)\.
- MERaLiON Team \(2024\)MERaLiON\-AudioLLM: bridging audio and language with large language models\.arXiv preprint arXiv:2412\.09818\.Cited by:[§1](https://arxiv.org/html/2607.10092#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.10092#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2607.10092#S2.SS3.p1.1)\.
- Mistral AI \(2025\)Voxtral\.arXiv preprint arXiv:2507\.13264\.Cited by:[§2\.1](https://arxiv.org/html/2607.10092#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.10092#S3.SS1.p2.1)\.
- J\. Peng, Y\. Wang, B\. Li, Y\. Guo, H\. Wang, Y\. Fang, Y\. Xi, H\. Li, X\. Li, K\. Zhang, S\. Wang, and K\. Yu \(2024\)A survey on speech large language models for understanding\.arXiv preprint arXiv:2410\.18908\.Cited by:[§1](https://arxiv.org/html/2607.10092#S1.p2.1)\.
- T\. N\. Quang, C\. Y\. L\. Won, M\. D\. Pham, Y\. He, S\. Sun, and A\. T\. Aw \(2026\)Direct preference optimization for English\-Mandarin code\-switching speech recognition in audio LLMs\.arXiv preprint arXiv:2605\.23975\.Cited by:[§2\.3](https://arxiv.org/html/2607.10092#S2.SS3.p2.1)\.
- A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever \(2022\)Robust speech recognition via large\-scale weak supervision\.arXiv preprint arXiv:2212\.04356\.Cited by:[§2\.1](https://arxiv.org/html/2607.10092#S2.SS1.p1.1)\.
- Y\. Tang and A\. K\. H\. Tung \(2023\)SQuAD\-SRC: a dataset for multi\-accent spoken reading comprehension\.InProceedings of the Thirty\-Second International Joint Conference on Artificial Intelligence \(IJCAI\-23\),pp\. 5206–5214\.Cited by:[§2\.2](https://arxiv.org/html/2607.10092#S2.SS2.p1.1)\.
- B\. Wang, X\. Zou, G\. Lin, S\. Sun, Z\. Liu, W\. Zhang, Z\. Liu, A\. Aw, and N\. F\. Chen \(2025a\)AudioBench: a universal benchmark for audio large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics \(NAACL\),Cited by:[§3\.1](https://arxiv.org/html/2607.10092#S3.SS1.p2.1)\.
- B\. Wang, X\. Zou, S\. Sun, W\. Zhang, Y\. He, Z\. Liu, C\. Wei, N\. F\. Chen, and A\. Aw \(2025b\)Advancing Singlish understanding: bridging the gap with datasets and multimodal models\.arXiv preprint arXiv:2501\.01034\.Cited by:[§2\.2](https://arxiv.org/html/2607.10092#S2.SS2.p1.1)\.
- H\. Zhu, L\. Ye, W\. Kang, Z\. Yao, L\. Guo, F\. Kuang, Z\. Han, W\. Zhuang, L\. Lin, and D\. Povey \(2026\)OmniVoice: towards omnilingual zero\-shot text\-to\-speech with diffusion language models\.arXiv preprint arXiv:2604\.00688\.Cited by:[§3\.2\.1](https://arxiv.org/html/2607.10092#S3.SS2.SSS1.p1.1)\.

## Appendix AList of Benchmarking Datasets

The below are the list of benchmarking datasets used, arranged in tasks mentioned:

1. 1\.ASR\-EN:MNSC/ASR\-Part1\-Test
2. 2\.ASR\-ZH:aishell1\_mandarin\_test,aishell3\_mandarin\_test,fleurs\_mandarin\_test
3. 3\.ASR\-ZH\-CS:seame\_dev\_man,seame\_dev\_sge
4. 4\.ASR\-MS:fleurs\_malay\_test
5. 5\.ASR\-MS\-CS:mesolitica\_malay\_test
6. 6\.ASR\-TA:fleurs\_tamil\_test,slr65\_tamil\_test,slr127\_tamil\_test
7. 7\.AR:MNSC/PQA\-AR\-Dialogue\-Test
8. 8\.GR:MNSC/PQA\-GR\-Dialogue\-Test
9. 9\.Spoken QA:internally developed, covering Home Team context across the four Singaporean languages
10. 10\.Speech QA:AudioLLMs/public\_sg\_speech\_qa\_test