To What Extent Do Large Language Models Understand Bangla Idioms?
Summary
This paper introduces a benchmark dataset for Bangla idioms and evaluates recent large language models on idiom-related tasks, revealing substantial variability in performance across models.
View Cached Full Text
Cached at: 09/04/26, 05:59 AM
# To What Extent Do Large Language Models Understand Bangla Idioms? Source: [https://arxiv.org/abs/2609.03410](https://arxiv.org/abs/2609.03410) [View PDF](https://arxiv.org/pdf/2609.03410) > Abstract:Idiomatic expressions are an integral part of natural language, reflecting cultural nuances and posing unique challenges for computational models, particularly in low\-resource languages\. In this paper, we present the first large\-scale benchmark dataset of Bangla idioms, complemented by a synthetic multiple\-choice question \(MCQ\) dataset for idiom meaning identification\. We conduct a comprehensive evaluation of recent large language models \(LLMs\) across three idiom\-related tasks: paraphrasing, idiom span detection, and meaning identification, leveraging zero\-shot and few\-shot prompting strategies\. Our results reveal substantial variability in model performance, with no single LLM consistently outperforming others across all tasks\. Notably, Phi\-4\-mini\-instruct excels in paraphrasing, Kimi\-K2\-32b\-instruct in span detection, and Gemini\-2\.5\-flash in meaning identification\. We believe that our datasets and analyses will provide valuable resources to guide future research in improving LLM comprehension of idiomatic expressions, particularly in Bangla and other low\-resource languages\. ## Submission history From: Mousumi Akter \[[view email](https://arxiv.org/show-email/819e642f/2609.03410)\] **\[v1\]**Thu, 3 Sep 2026 06:15:15 UTC \(552 KB\)
Similar Articles
IdiomX A Multilingual Benchmark for Idiom Understanding, Retrieval, and Interpretation
IdiomX is a large-scale multilingual benchmark for idiom understanding, retrieval, and interpretation, containing over 190K examples across English, Arabic, and French, with four tasks for evaluating language models on idiomatic expressions.
When English Rewrites Local Knowledge: Global Narrative Dominance in Large Language Models
This paper introduces CulturalNB, a dataset of Bengali cultural question-answer pairs, and evaluates nine LLMs for cross-lingual cultural bias. Findings show that English prompting increases global narrative substitution and reduces local perspectives, revealing that cultural failures in LLMs are grounding and prioritization issues, not just missing knowledge.
VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP
This paper introduces VIVID, the first systematic benchmark for evaluating culturally grounded figurative language understanding in Vietnamese, comprising 1,636 idioms and proverbs. Evaluation of eight state-of-the-art models reveals significant gaps, with Vietnamese-specialized models drastically underperforming multilingual systems and even top models achieving less than 50% correctness on average.
Can We Understand How Large Language Models Reason?
This article explores the ongoing efforts and challenges in understanding how large language models reason, focusing on interpretability research.
Poly-Dialectal Neural Machine Translation System for Bangla Regional Dialects
This paper presents a unified poly-dialectal neural machine translation system for 12 Bangla regional dialects, introducing the largest multi-dialect parallel corpus to date and achieving state-of-the-art BLEU scores with a fine-tuned BanglaT5 model using DoRA.