Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages

# Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
Source: [https://arxiv.org/html/2605.17152](https://arxiv.org/html/2605.17152)
Firoj Alam, Shammur Absar Chowdhury, Enamul Hoque Prince Qatar Computing Research Institute, HBKU, Qatar, York University, Canada \{fialam, shchowdhury\}@hbku\.edu\.qa, enamulh@yorku\.ca [https://mm\-llms\-in\-the\-wild\.github\.io](https://mm-llms-in-the-wild.github.io/)

Multilingual and Multimodal LLMs in the Wild: Building for Low\-Resource Languages

Firoj Alam, Shammur Absar Chowdhury, Enamul Hoque PrinceQatar Computing Research Institute, HBKU, Qatar, York University, Canada\{fialam, shchowdhury\}@hbku\.edu\.qa, enamulh@yorku\.ca[https://mm\-llms\-in\-the\-wild\.github\.io](https://mm-llms-in-the-wild.github.io/)

## 1Abstract

Multimodal LLMs are evolving from vision–language to tri\-modality that*see, hear, and read*, yet pipelines and benchmarks remain English\-centric and compute\-heavy\. The tutorial offers an overview of this emerging research area for*multilingual**multimodality*acrosstext, speech, and visionunder limited data/compute budgets, synthesizing foundations, recent multilingual models \(PALO, Maya\), speech–text LLMs\. We cover low\-cost data creation/curation; adapter stacks for tri\-modal alignment;*culture\-aware*evaluation beyond English and hands on resources for fine\-tuning a compact multilingual VLM and wiring a speech→\\rightarrowtext→\\rightarrowLLM pipeline\. The content will be delivered as an interactive half\-day tutorial, designed for researchers and practitioners working on multilingual, multimodal AI in low\-resource language settings\.

## 2Introduction

Multimodal large language models \(LMMs\) are transforming how we build AI systems: instead of only reading text, they can now*see*,*hear*, and*read*—integrating visual, audio, and textual signals under a unified architectureAlamet al\.\([2025](https://arxiv.org/html/2605.17152#bib.bib22)\)\. Surveys show that while LMMs are advancing rapidly, most datasets, benchmarks, and models remain heavily English\-centric and optimized for high\-resource languages and settingsYin and Fu \([2024](https://arxiv.org/html/2605.17152#bib.bib9)\); Wuet al\.\([2023](https://arxiv.org/html/2605.17152#bib.bib8)\)\. This leaves many low\-resource languages, dialects, and culturally\-specific modalities under\-served, especially in multilingual, speech\-and\-vision contexts\.

The multilingual and multi\-modal AI community – spanning NLP, speech processing, and computer vision, stands at a critical juncture\. On one hand, we have foundation models \(e\.g\., vision–language systems\) that demonstrate strong capabilities in high\-resource contexts; on the other, communities working with under\-represented languages lack tools, datasets, and evaluation protocols that support tri\-modal workflows \(text, speech, vision\) in resource\-constrained environments\. Recent work \(e\.g\., PaLI, mSLAM\) shows promise for multilingual cross\-modal pre\-trainingChenet al\.\([2022](https://arxiv.org/html/2605.17152#bib.bib5)\); Bapnaet al\.\([2022](https://arxiv.org/html/2605.17152#bib.bib6)\), but large gaps remain in end\-to\-end pipelines, culturally grounded benchmarks, and efficient architectures tailored for real\-world low\-resource deployment\.

In this tutorial we aim to bring together the state\-of\-the\-art in multilingual multimodal LMMs with practical, reproducible recipes for data creation \(covering, text, speech, vision from different disciplines\), model alignment, fine\-tuning, and evaluation, focusing explicitly on low\-resource languages and cultural contexts\. We will cover foundational models and architectures, examine evaluation and bias considerations, and provide hands\-on labs for fine\-tuning compact multilingual vision–language models and wiring speech → text → LLM pipelines\. Our goal is to equip researchers and engineers working in multilingual, multi\-modal AI with actionable tools, checklists, and benchmarks to deliver inclusive, grounded AI systems beyond the high\-resource norm\.

### 2\.1Foundations of Multimodal and Multilingual Models

We review the evolution of large models: from text\-only LLMs based on the Transformer architectureVaswaniet al\.\([2017](https://arxiv.org/html/2605.17152#bib.bib206)\), to vision\-language models \(e\.g\., BLIP\-2, LLaVA\) and now to unified speech–text–vision systems \(e\.g\., PaLM\-E, AudioPaLM\)\. We highlight how multilingual extensions of these, while nascent, open pathways for inclusive AI in under\-represented languages\. We also discuss how modalities add complementary information, e\.g\., visual or audio context can compensate for scant textual data in low\-resource languagesLupascuet al\.\([2025](https://arxiv.org/html/2605.17152#bib.bib7)\)\.

### 2\.2Challenges in Low\-Resource Multimodality

Lower\-resource languages bring numerous obstacles: scarcity of paired text/image/audio data, missing benchmarks, disparate dialects/scripts, and compute constraints for large models\. For instance, a survey found that out of 106 studies in low\-resource multimodal learning across 75 languages, the vast majority focused on text\+image pairs but neglected audio or video modalitiesLupascuet al\.\([2025](https://arxiv.org/html/2605.17152#bib.bib7)\)\. We analyse key bottlenecks, data creation, modality alignment, adaptation of pre\-trained models, and evaluation frameworks, and how these hinder deployment in real\-world multilingual/multimodal settings\.

### 2\.3Dialectal Challenge

Collecting dialectal multimodal data at scale required tackling issues that standard VQA corpora rarely face\. In*EverydayMMQA*/OASIS, we had to\(i\)recruit and balance native speakers across 18 countries and Arabic varieties;\(ii\)capture*spoken*questions that naturally include dialectal phonology, code\-switching, and region\-specific lexicon;\(iii\)align speech/text prompts with culturally grounded images beyond object labels; and\(iv\)verify answers under divergent orthographies and local conventions\. On the evaluation side, our benchmarks emphasize pragmatic, commonsense, and culturally aware aspects and, across four input modes \(speech\-only, text\-only, speech\+image, text\+image\), surface failure modes typical of dialectal settings \(e\.g\., misinterpretation of dialectal terms, sensitivity to ASR noise, weaker transfer from English\)\. Overall, the study highlights significant gaps between current general\-purpose multimodal LLMs and the demands of dialect\-rich, everyday queries, and motivates training and evaluation pipelines explicitly grounded in local culture\.

### 2\.4Relevance for Our Community

This tutorial is timely for researchers and practitioners working at the intersection of multilingual NLP, speech processing, and vision\. Its relevance spans:

- •Building multimodal datasets and models for under\-served languages \(e\.g\., Arabic dialects, Indic, African, Southeast Asian languages\)\.
- •Adapting large vision–language–speech models with limited data and compute via techniques like PEFT, adapters, and MoE\.
- •Evaluating models beyond English: culture\-aware benchmarks, dialect resilience, and modality robustness\.
- •Deploying inclusive and accessible systems, with awareness of language diversity, script/dialect variation, and societal context\.

### 2\.5What this tutorial offers

This tutorial guides participants through building inclusive, multilingual, multimodal systems \(text, speech & vision\) for low\-resource settings\. We begin with the evolution of LLMs, covering their architecture, multilingual extensions, and multimodal variants\. Then we explore concrete model families: vision–language \(e\.g\., PALO\), speech–text LMMs \(SeamlessM4T, AudioPaLM\), and multilingual multimodal models \(e\.g\., Maya\)\. Next, we dive into hands\-on methods: low\-cost data creation and curation \(translation, weak supervision, filtering\); efficient training \(PEFT, adapters, Mixture\-of\-Experts\); and tri\-modal alignment workflows\. We then examine evaluation and deployment: multilingual cultural benchmarks \(xGQA, HaVQA\), visualization datasets \(FigureQA, CharXiv\), dialectal tests, and addressing hallucination, bias, toxicity, as well as compute/latency trade\-offs\. An outline of the tutorial is reported in Section[6](https://arxiv.org/html/2605.17152#S6)\.

## 3Type of the Tutorial

The tutorial is both introductory, covering a number of topics related to the capabilities of LLMs, but it is also cutting\-edge, covering some latest developments in these areas\. Attendees will have an overview of tasks, languages, dialects and modalities related to LLMs, which will put them up to speed to do research in the area\. The tutorial targets anyone interested in employing LLMs for NLP, speech and multimodal tasks\. We believe researchers working on lower\-resource languages will be especially interested\.

## 4Tutorial organisers

- •[Firoj Alam](http://sites.google.com/site/firojalam/)is a Scientist at the Qatar Computing Research Institute \(QCRI\), HBKU\.
- •
- •[Enamul Hoque](https://www.yorku.ca/enamulh/)is an Associate Professor at York University\.

## 5Target Audience

Researchers and practitioners in NLP, speech, and vision, especially those building for low\-resource languages, dialects, or culturally grounded domains\. Suitable for graduate students, academic/industry ML engineers, and dataset curators with basic LLM fine\-tuning experience\. Minimal prior speech or vision background is required; we provide extensive materials\. Attendees seeking practical PEFT recipes multimodal adapters, and robust, culture\-aware evaluation will benefit most\.

## 6Outline

Below, we offer an outline of the tutorial\. More information and materials will be available online on the tutorial website upon acceptance\.

### 6\.1Introduction

1. \(i\)Why tri\-modality \(text–vision–speech\) matters for low\-resource & Global South contexts
2. \(ii\)From vision–language to general\-purpose multimodality: BLIP\-2, LLaVA, KOSMOS\-1, PaLM\-E
3. \(iii\)What this tutorial adds: multilingual focus, speech integration, efficiency \(PEFT/MoE\), culture\-aware evaluation

*References:*Liet al\.\([2023](https://arxiv.org/html/2605.17152#bib.bib10)\); Liuet al\.\([2023](https://arxiv.org/html/2605.17152#bib.bib11)\); Huanget al\.\([2023](https://arxiv.org/html/2605.17152#bib.bib13)\); Driesset al\.\([2023](https://arxiv.org/html/2605.17152#bib.bib23)\)

### 6\.2Multilingual and Multimodal Models

The following are representative models; we will cover additional systems in the tutorial\.

1. \(i\)Vision–language \(multilingual\) 1. \(a\)PALO \(10\-language LMM\), Maya \(8\-language, toxicity\-aware data pipeline\)
2. \(ii\)Speech–text LMMs 1. \(a\)SeamlessM4T \(unified S2S/S2T/T2S/T2T/ASR\), AudioPaLM \(joint speech\+text\)
3. \(iii\)Robust ASR backbones for pipelines 1. \(a\)Whisper \(weakly supervised, multilingual\)

*References:*Rasheedet al\.\([2024](https://arxiv.org/html/2605.17152#bib.bib14)\); Alamet al\.\([2024b](https://arxiv.org/html/2605.17152#bib.bib15)\); SEAMLESS Communication Team and Others \([2025](https://arxiv.org/html/2605.17152#bib.bib16)\); Rubensteinet al\.\([2023](https://arxiv.org/html/2605.17152#bib.bib17)\); Radfordet al\.\([2022](https://arxiv.org/html/2605.17152#bib.bib225)\)

### 6\.3Multilingual & Multimodal Resource Development

#### 6\.3\.1Multilingual & Multimodal Resources

1. \(i\)Low\-cost pipelines: translation & back\-translation, weak supervision, OCR/ASR bootstraps
2. \(ii\)Safety & culture: toxicity filtering, demographic balance, licensing/consent \(Maya case study\)
3. \(iii\)Resource orientation: multilingual V\+L evaluation sets \(xGQA, MaRVL, HaVQA\) and training data considerations

*References:*Alamet al\.\([2025](https://arxiv.org/html/2605.17152#bib.bib22),[2024b](https://arxiv.org/html/2605.17152#bib.bib15)\); Pfeifferet al\.\([2022](https://arxiv.org/html/2605.17152#bib.bib19)\); Liuet al\.\([2021](https://arxiv.org/html/2605.17152#bib.bib20)\); Paridaet al\.\([2023](https://arxiv.org/html/2605.17152#bib.bib21)\)

#### 6\.3\.2Reasoning Across Visual and Structured Modalities

1. \(i\)Visualization datasets \(FigureQA, CharXiv, ChartQAPro, DashboardQA\)\)
2. \(ii\)Reasoning Techniques: multimodal chain\-of\-thought, ReAct prompting, and structured decoding for spatial/tabular data\.

*References:*Kahouet al\.\([2018](https://arxiv.org/html/2605.17152#bib.bib1)\); Wanget al\.\([2024](https://arxiv.org/html/2605.17152#bib.bib2)\); Masryet al\.\([2025](https://arxiv.org/html/2605.17152#bib.bib3)\); Karthaet al\.\([2026](https://arxiv.org/html/2605.17152#bib.bib4)\)

### 6\.4Architectures and Efficient Training

1. \(i\)Adapter/projector stacks for VLMs \(e\.g\., BLIP\-2 Q\-Former\), early vs\. late fusion
2. \(ii\)PEFT in practice \(LoRA/QLoRA\), quantization notes for constrained VRAM
3. \(iii\)Mixture\-of\-Experts for modality/language specialization: MoME, Uni\-MoE; routing & capacity factors

*References:*Liet al\.\([2023](https://arxiv.org/html/2605.17152#bib.bib10)\); Shenet al\.\([2024](https://arxiv.org/html/2605.17152#bib.bib18)\); Liet al\.\([2025](https://arxiv.org/html/2605.17152#bib.bib24)\)

### 6\.5Speech\-centric LLMs

1. \(i\)Wiring speech→\\rightarrowtext→\\rightarrowLLM for VQA/QA; streaming, VAD/diarization hooks
2. \(ii\)Unified speech–text LMMs vs\. cascades: deployment trade\-offs \(latency, robustness, coverage\)

*References:*Radfordet al\.\([2022](https://arxiv.org/html/2605.17152#bib.bib225)\); SEAMLESS Communication Team and Others \([2025](https://arxiv.org/html/2605.17152#bib.bib16)\); Rubensteinet al\.\([2023](https://arxiv.org/html/2605.17152#bib.bib17)\)

### 6\.6Evaluation, Benchmarking Resources, Error analysis

1. \(i\)Culture\-aware, multilingual benchmarks and diagnostics \(xGQA, MaRVL, HaVQA\)
2. \(ii\)Stress tests: dialect shifts, noise/occlusion, OCR\-heavy inputs, hallucination & grounding checks

*References:*Pfeifferet al\.\([2022](https://arxiv.org/html/2605.17152#bib.bib19)\); Liuet al\.\([2021](https://arxiv.org/html/2605.17152#bib.bib20)\); Paridaet al\.\([2023](https://arxiv.org/html/2605.17152#bib.bib21)\)

### 6\.7Resources and demo applications & wrap\-up \[25 min\]

1. \(i\)LoRA\-tune a compact multilingual VLM; quick eval on a culture\-aware slice
2. \(ii\)Speech front\-end \(Whisper/Seamless\) into an instruction\-tuned LLM; measure ASR→\\rightarrowtask impact

*References:*Liuet al\.\([2023](https://arxiv.org/html/2605.17152#bib.bib11)\); SEAMLESS Communication Team and Others \([2025](https://arxiv.org/html/2605.17152#bib.bib16)\)

## 7Technical Requirements

No special requirements; we will use standard A/V setups provided by the organizers\.

## 8Diversity Considerations

We will run an inclusive tutorial that reflects linguistic, cultural, geographic, and disciplinary diversity\.

Contribution to academic diversity\.The tutorial centers multilingual, multimodal*everyday knowledge*and explicitly links language, speech, and vision communities\. We foreground low\-resource and culturally specific contexts and encourage collaboration across academia, industry, and public\-interest groups\.

Representation\.We will actively advertise the tutorial globally, with special outreach to under\-represented regions and communities, to ensure participation from all corners of the world\. Our outreach will include different mailing lists, and social media\.

## 9Reading List

Relevant papers has been listed in Section[6](https://arxiv.org/html/2605.17152#S6)\.

## 10Presenters

##### [Firoj Alam](http://sites.google.com/site/firojalam/publications)

\(Qatar Computing Research Institute, HBKU\) is a Senior Scientist and a senior IEEE and ACM member\. He has been co\-organizing several workshops and shared tasks, including BLP\-2023 and GenAI Content Detection at COLING\-2025\. His shared task experience includes the CheckThat\! Lab at CLEF \(2021\-2025\), NLP4IF 2021, and SemEval 2021 Task 6\. He has co\-oranized a tutorial at EACL\-2024\. He has also served as a PC and SPC member for numerous conferences \(∗ACL, AAAI, IJCAI, ICWSM, etc\.\), as well as a publications and panel chair and has volunteered as a reviewer for various journals\.

##### [Shammur Absar Chowdhury](http://shammur.one/)

\(Qatar Computing Research Institute, HBKU\) is a scientist\. She broadly works in the area of Conversational AI encompassing language acquisition, speech discourse, code\-switching speech recognition and turn\-taking in a spontaneous human conversation\. Shammur has been contributing to strengthen the community by organizing workshops and shared tasks like IEEE SLT Workshop 2022 \(local chair\), JSALT Summer workshop 2022 \(Multilingual and CS ASR\), SemEval\-2022 \(Task3\) among others\. She has serve as PC and SPC member of many top NLP and Speech conferences – ACL, Interspeech, ICASSP, NAACL, EMNLP etc; and volunteered as a reviewer for various international journals\.

##### [Enamul Hoque Prince](https://www.yorku.ca/enamulh)

\(York University\) is an Associate Professor, and Director of the Intelligent Visualization Lab\. His research brings together information visualization, natural language processing, and human–computer interaction to make data exploration more accessible, inclusive, and responsible\. His recent work focuses on multimodal large language models and agentic frameworks for data visualization and analytics, leading to widely used benchmarks and models\. He serves the community as an Area Chair for the ACL Rolling Review and on the IEEE VIS program committee\.

## 11Other Information

At EACL 2024, we organized a closely related tutorialAlamet al\.\([2024a](https://arxiv.org/html/2605.17152#bib.bib25)\)that drew 354 registrants, with 40\+ live attendees \(in person and online\)\. We have also run Birds\-of\-a\-Feather sessions at EMNLP and COLING on similar themes, attracting 30–50 participants per session\. Accordingly, we expect 40\+ attendees for this edition\.

## 12Ethics Statement

Our tutorial is based on our own work in the area, related studies and public sources\. Credit will be given wherever needed\. Any biases are unintended\.

## References

- LLMs for low resource languages in multilingual, multimodal and dialectal settings\.InProceedings of the 18th conference of the European chapter of the association for computational linguistics: tutorial abstracts,pp\. 27–33\.Cited by:[§11](https://arxiv.org/html/2605.17152#S11.p1.1)\.
- F\. Alam, A\. E\. Shahroor, M\. A\. Hasan, Z\. S\. Ali, H\. H\. Bhatti, M\. B\. Kmainasi, S\. A\. Chowdhury, B\. Mousi, F\. Dalvi, N\. Durrani,et al\.\(2025\)EverydayMMQA: a multilingual and multimodal framework for culturally grounded spoken visual qa\.arXiv preprint arXiv:2510\.06371\.Cited by:[§2](https://arxiv.org/html/2605.17152#S2.p1.1),[§6\.3\.1](https://arxiv.org/html/2605.17152#S6.SS3.SSS1.p1.1)\.
- N\. Alam, K\. R\. Kanjula, S\. Guthikonda,et al\.\(2024b\)Maya: an instruction finetuned multilingual multimodal model\.arXiv preprint arXiv:2412\.07112\.External Links:[Link](https://arxiv.org/abs/2412.07112)Cited by:[§6\.2](https://arxiv.org/html/2605.17152#S6.SS2.p1.2),[§6\.3\.1](https://arxiv.org/html/2605.17152#S6.SS3.SSS1.p1.1)\.
- A\. Bapna, C\. Cherry, Y\. Zhang, Y\. Jia, M\. Johnson, Y\. Cheng, S\. Khanuja, J\. Riesa, and A\. Conneau \(2022\)mSLAM: massively multilingual joint pre\-training for speech and text\.arXiv e\-prints,pp\. arXiv–2202\.Cited by:[§2](https://arxiv.org/html/2605.17152#S2.p2.1)\.
- X\. Chen, X\. Wang, S\. Changpinyo, P\. Padlewski, D\. Salz, S\. Goodman, A\. Grycner, B\. Mustafa, L\. Beyer,et al\.\(2022\)PaLI: a jointly\-scaled multilingual language\-image model\.arXiv preprint arXiv:2209\.06794\.Cited by:[§2](https://arxiv.org/html/2605.17152#S2.p2.1)\.
- D\. Driess, F\. Xia, M\. S\. M\. Sajjadi, and et al\. \(2023\)PaLM\-E: an embodied multimodal language model\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 8469–8488\.External Links:[Link](https://proceedings.mlr.press/v202/driess23a.html)Cited by:[§6\.1](https://arxiv.org/html/2605.17152#S6.SS1.p1.1)\.
- S\. Huang, L\. Dong, W\. Wang,et al\.\(2023\)Language is not all you need: aligning perception with language models\.arXiv preprint arXiv:2302\.14045\.External Links:[Link](https://arxiv.org/abs/2302.14045)Cited by:[§6\.1](https://arxiv.org/html/2605.17152#S6.SS1.p1.1)\.
- S\. E\. Kahou, V\. Michalski, A\. Atkinson, Á\. Kádár, A\. Trischler, and Y\. Bengio \(2018\)FigureQA: an annotated figure dataset for visual reasoning\.In6th International Conference on Learning Representations \(ICLR 2018\), Workshop Track Proceedings,External Links:[Link](https://openreview.net/forum?id=H1mz0OyDz)Cited by:[§6\.3\.2](https://arxiv.org/html/2605.17152#S6.SS3.SSS2.p1.1)\.
- A\. Kartha, A\. Masry, M\. S\. Islam, T\. Lang, S\. Rahman, R\. Mahbub, M\. Rahman, M\. Ahmed, M\. R\. Parvez, E\. Hoque, and S\. Joty \(2026\)DashboardQA: benchmarking multimodal agents for question answering on interactive dashboards\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 3385–3407\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.177),[Link](https://aclanthology.org/2026.findings-eacl.177/)Cited by:[§6\.3\.2](https://arxiv.org/html/2605.17152#S6.SS3.SSS2.p1.1)\.
- J\. Li, D\. Li, S\. Savarese, and S\. Hoi \(2023\)BLIP\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 19730–19742\.External Links:[Link](https://proceedings.mlr.press/v202/li23q.html)Cited by:[§6\.1](https://arxiv.org/html/2605.17152#S6.SS1.p1.1),[§6\.4](https://arxiv.org/html/2605.17152#S6.SS4.p1.1)\.
- Y\. Li, S\. Jiang, B\. Hu, L\. Wang, W\. Zhong, W\. Luo, L\. Ma, and M\. Zhang \(2025\)Uni\-MoE: scaling unified multimodal llms with mixture of experts\.IEEE Transactions on Pattern Analysis and Machine Intelligence47\(5\),pp\. 3424–3439\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2025.3532688),[Link](https://www.computer.org/csdl/journal/tp/2025/05/10887014/24j49GWBANq)Cited by:[§6\.4](https://arxiv.org/html/2605.17152#S6.SS4.p1.1)\.
- F\. Liu, E\. Bugliarello, E\. M\. Ponti, S\. Reddy, N\. Collier, and D\. Elliott \(2021\)Visually grounded reasoning across languages and cultures\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,Online and Punta Cana, Dominican Republic,pp\. 10467–10485\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.818),[Link](https://aclanthology.org/2021.emnlp-main.818/)Cited by:[§6\.3\.1](https://arxiv.org/html/2605.17152#S6.SS3.SSS1.p1.1),[§6\.6](https://arxiv.org/html/2605.17152#S6.SS6.p1.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\),External Links:[Link](https://papers.nips.cc/paper_files/paper/2023/file/6dcf277ea32ce3288914faf369fe6de0-Paper-Conference.pdf)Cited by:[§6\.1](https://arxiv.org/html/2605.17152#S6.SS1.p1.1),[§6\.7](https://arxiv.org/html/2605.17152#S6.SS7.p1.1)\.
- M\. Lupascu, A\. Rogoz, M\. S\. Stupariu, and R\. T\. Ionescu \(2025\)Large multimodal models for low\-resource languages: a survey\.arXiv preprint arXiv:2502\.05568\.Cited by:[§2\.1](https://arxiv.org/html/2605.17152#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2605.17152#S2.SS2.p1.1)\.
- A\. Masry, M\. S\. Islam, M\. Ahmed, A\. Bajaj, F\. Kabir, A\. Kartha, M\. T\. R\. Laskar, M\. Rahman, S\. Rahman, M\. Shahmohammadi, M\. Thakkar, M\. R\. Parvez, E\. Hoque, and S\. Joty \(2025\)ChartQAPro: a more diverse and challenging benchmark for chart question answering\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 19123–19151\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.978),[Link](https://aclanthology.org/2025.findings-acl.978/)Cited by:[§6\.3\.2](https://arxiv.org/html/2605.17152#S6.SS3.SSS2.p1.1)\.
- S\. Parida, I\. Abdulmumin, S\. H\. Muhammad, A\. Bose, G\. S\. Kohli, I\. Ahmad, K\. Kotwal, S\. Deb Sarkar, O\. Bojar, and H\. Kakudi \(2023\)HaVQA: a dataset for visual question answering and multimodal research in hausa language\.InFindings of the Association for Computational Linguistics \(ACL 2023\),Toronto, Canada,pp\. 10162–10183\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.646),[Link](https://aclanthology.org/2023.findings-acl.646/)Cited by:[§6\.3\.1](https://arxiv.org/html/2605.17152#S6.SS3.SSS1.p1.1),[§6\.6](https://arxiv.org/html/2605.17152#S6.SS6.p1.1)\.
- J\. Pfeiffer, G\. Geigle, A\. Kamath, J\. O\. Steitz, S\. Roth, I\. Vulić, and I\. Gurevych \(2022\)xGQA: cross\-lingual visual question answering\.InFindings of the Association for Computational Linguistics \(ACL 2022\),Dublin, Ireland,pp\. 2497–2511\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.196),[Link](https://aclanthology.org/2022.findings-acl.196/)Cited by:[§6\.3\.1](https://arxiv.org/html/2605.17152#S6.SS3.SSS1.p1.1),[§6\.6](https://arxiv.org/html/2605.17152#S6.SS6.p1.1)\.
- A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever \(2022\)Robust speech recognition via large\-scale weak supervision\.arXiv preprint arXiv:2212\.04356\.Cited by:[§6\.2](https://arxiv.org/html/2605.17152#S6.SS2.p1.2),[§6\.5](https://arxiv.org/html/2605.17152#S6.SS5.p1.1)\.
- H\. Rasheed, M\. Maaz, A\. Shaker, S\. Khan, H\. Cholakkal, R\. M\. Anwer, T\. Baldwin, M\. Felsberg, and F\. S\. Khan \(2024\)PALO: a polyglot large multimodal model for 5b people\.arXiv preprint arXiv:2402\.14818\.External Links:[Link](https://arxiv.org/abs/2402.14818)Cited by:[§6\.2](https://arxiv.org/html/2605.17152#S6.SS2.p1.2)\.
- P\. K\. Rubenstein, C\. Asawaroengchai, D\. D\. Nguyen,et al\.\(2023\)AudioPaLM: a large language model that can speak and listen\.arXiv preprint arXiv:2306\.12925\.External Links:[Link](https://arxiv.org/abs/2306.12925)Cited by:[§6\.2](https://arxiv.org/html/2605.17152#S6.SS2.p1.2),[§6\.5](https://arxiv.org/html/2605.17152#S6.SS5.p1.1)\.
- SEAMLESS Communication Team and Others \(2025\)Joint speech and text machine translation for up to 100 languages\.Nature637,pp\. 587–593\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-08359-z),[Link](https://www.nature.com/articles/s41586-024-08359-z)Cited by:[§6\.2](https://arxiv.org/html/2605.17152#S6.SS2.p1.2),[§6\.5](https://arxiv.org/html/2605.17152#S6.SS5.p1.1),[§6\.7](https://arxiv.org/html/2605.17152#S6.SS7.p1.1)\.
- L\. Shen, G\. Chen, R\. Shao, W\. Guan, and L\. Nie \(2024\)MoME: mixture of multimodal experts for generalist multimodal large language models\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS 2024\),External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/4a3a14b9536806a0522930007c5512f7-Paper-Conference.pdf)Cited by:[§6\.4](https://arxiv.org/html/2605.17152#S6.SS4.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§2\.1](https://arxiv.org/html/2605.17152#S2.SS1.p1.1)\.
- Z\. Wang, M\. Xia, L\. He, H\. Chen, Y\. Liu, R\. Zhu, K\. Liang, X\. Wu, H\. Liu, S\. Malladi, A\. Chevalier, S\. Arora, and D\. Chen \(2024\)CharXiv: charting gaps in realistic chart understanding in multimodal llms\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/cdf6f8e9fd9aeaf79b6024caec24f15b-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§6\.3\.2](https://arxiv.org/html/2605.17152#S6.SS3.SSS2.p1.1)\.
- J\. Wu, W\. Gan, Z\. Chen, S\. Wan, and P\. S\. Yu \(2023\)Multimodal large language models: a survey\.arXiv preprint arXiv:2311\.13165\.External Links:[Link](https://arxiv.org/abs/2311.13165)Cited by:[§2](https://arxiv.org/html/2605.17152#S2.p1.1)\.
- S\. Yin and C\. Fu \(2024\)A survey on multimodal large language models\.National Science Review11\(12\),pp\. nwae403\.External Links:[Document](https://dx.doi.org/10.1093/nsr/nwae403),[Link](https://academic.oup.com/nsr/article/11/12/nwae403/7896414)Cited by:[§2](https://arxiv.org/html/2605.17152#S2.p1.1)\.