VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP

# VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP
Source: [https://arxiv.org/html/2608.03095](https://arxiv.org/html/2608.03095)
###### Abstract

We present VIVID \(Vietnamese Idioms for Validation and Interpretation Depth\), the first systematic benchmark for evaluating culturally grounded figurative language understanding in Vietnamese\. VIVID comprises 1,636 idioms and proverbs annotated with five complexity traits \(literal expressions, pragmatic nuances, Sino\-Vietnamese terms, uncommon vocabulary, folk knowledge\) and seven semantic themes\. We establish an evaluation framework combining generative and discriminative tasks, proposing an LLM\-as\-a\-Judge approach with aspect\-based prompting validated against human judgment \(Cohen’s κ = 0\.792\)\. Evaluating eight state\-of\-the\-art models reveals critical gaps: Vietnamese\-specialized models drastically underperform multilingual systems \(VinaLLaMA\-7B: 0\.13 vs\. GPT\-4o: 2\.46\), and even top models achieve less than 50% correctness on average\. Notably, few\-shot prompting does not universally improve performance, with GPT\-4o exhibiting degradation due to stylistic overfitting\. Our analysis exposes systematic failures including literal over\-interpretation, lexical gaps, and pragmatic flattening, demonstrating that current models lack cultural competence for nuanced figurative interpretation\. VIVID provides an essential tool for advancing figurative language understanding in culturally rich contexts\. We release codes and datasets at[https://github\.com/ReML\-AI/VIVID](https://github.com/ReML-AI/VIVID)\.

Keywords:Vietnamese Benchmark, Figurative Language Evaluation, Idioms and Proverbs

\\NAT@set@cites

VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP

Tu Tran Do1∗, Nhat Ngoc Nguyen1∗, Khanh\-Tung Tran2Hoang D\. Nguyen2,Tu Minh Phuong1, Long Hoang Dang11Posts and Telecommunications Institute of Technology \(PTIT\), Vietnam2School of Computer Science and Information Technology, University College Cork, Ireland\{tudt\.b22kh109, nhatnn\.b22at208\}@stu\.ptit\.edu\.vn123128577@umail\.ucc\.ie hn@cs\.ucc\.ie\{phuongtm, longdh\}@ptit\.edu\.vnAbstract content

\\textsuperscript\{\\normalsize\*\}\\textsuperscript\{\\normalsize\*\}footnotetext:These authors contributed equally\.## 1\. Introduction

Figurative language understanding remains a critical challenge in natural language processing, requiring models to integrate linguistic knowledge, cultural context, and pragmatic reasoningLiuet al\.\([2022](https://arxiv.org/html/2608.03095#bib.bib37)\)\. In Vietnamese linguistics, an idiom is defined as a fixed multi\-word expression whose meaning cannot be straightforwardly inferred from the literal meanings of its constituent words\. A proverb, in contrast, is a short, often rhythmic sentence that encapsulates folk knowledge, life experience, or moral lessonsPhê \([2021](https://arxiv.org/html/2608.03095#bib.bib18)\)\. Vietnamese idioms and proverbs present unique interpretation difficulties through archaic Sino\-Vietnamese vocabulary, subtle pragmatic nuances \(sarcasm, irony\), and folk knowledge about traditional agricultural practices\. Despite Vietnamese having substantial NLP resources, including benchmarks for natural language inferenceHuynhet al\.\([2022](https://arxiv.org/html/2608.03095#bib.bib40)\), sentiment analysisHoet al\.\([2019](https://arxiv.org/html/2608.03095#bib.bib41)\), and multi\-hop reasoningLeet al\.\([2022a](https://arxiv.org/html/2608.03095#bib.bib42)\),no systematic evaluation framework exists for assessing culturally grounded figurative language understanding\.

This gap motivates two research questions\.First, how do state\-of\-the\-art language models perform on Vietnamese figurative language?Existing benchmarks focus on high\-resource languages like EnglishChakrabartyet al\.\([2022](https://arxiv.org/html/2608.03095#bib.bib21)\), KoreanWanget al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib25)\), or Ethiopian languagesAzimeet al\.\([2025](https://arxiv.org/html/2608.03095#bib.bib24)\)evaluating binary idiomaticity detection rather than cultural understanding\. Without a Vietnamese benchmark, we cannot assess whether Vietnamese\-specialized or multilingual models accurately interpret idioms, nor identify which linguistic features challenge these systems\.Second, how can we reliably evaluate figurative language understanding?Traditional metrics like BLEU fail to capture semantic nuancesTanget al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib27)\); Wanget al\.\([2025a](https://arxiv.org/html/2608.03095#bib.bib44)\), while human evaluation is expensive and unscalable\.

To address these questions, we introduceVIVID\(Vietnamese Idioms for Validation and Interpretation Depth\), a novel benchmark of 1,636 Vietnamese idioms and proverbs with dual\-layer annotations\. First, we develop a linguistically\-motivated taxonomy organizing expressions into three aspects \(Meaning, Lexical, Agricultural & Customary Knowledge\) with five complexity traits that capture primary interpretation challenges\. Second, we categorize idioms into seven semantic themes \(Criticism, Life Lessons, Work and Nature, Social Relationships, Love, Virtues, Other\) reflecting Vietnamese cultural discourse\. We evaluate eight state\-of\-the\-art models—including Vietnamese\-specialized systems \(Vistral\-7B, VinaLLaMA\-7B, GreenMind\-14B\) and large multilingual models \(GPT\-4o, Gemini Flash 2\.5, Llama\-4\-Scout, Qwen\-3,Llama\-SEA\-LION\-v3\)—across complementary evaluation tasks\.

For evaluation, we develop a comprehensive evaluation framework combining generative and discriminative approaches\. For generative evaluation, we utilize an LLM\-as\-a\-Judge framework where models generate idiom explanations evaluated by GPT\-4\.1\. Through systematic comparison of four prompting strategies, we establish that aspect\-based evaluation—explicitly considering factual accuracy, completeness, and native speaker alignment—achieves strongest human correlation \(Cohen’s κ = 0\.792\)\. For discriminative evaluation, we apply multiple\-choice classification tasks for topic and linguistic characteristic identification using exact\-match evaluationGaoet al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib29)\)\.

Our findings reveal critical gaps across all model types: Even state\-of\-the\-art models achieve less than 50% correctness on average, with significant variation across complexity traits—models handle Sino\-Vietnamese terms better than pragmatic nuances or folk knowledge\. Among comparable\-sized models, Vietnamese\-specialized systems \(GreenMind\-14B: 1\.09\) perform similarly to multilingual counterparts \(Qwen\-3\-14B: 1\.09\), suggesting that current language\-specific training does not provide advantages for cultural figurative understanding\. Notably, few\-shot prompting does not universally help: while benefiting most models, GPT\-4o shows degradation due to stylistic overfitting\.

Our contributions are threefold:\(1\)VIVID, the first systematic benchmark for Vietnamese figurative language with 1,636 validated idiom\-explanation pairs and dual\-layer annotations;\(2\)evaluation protocols combining a novel LLM\-as\-a\-Judge framework for generative assessment with discriminative classification tasks; and\(3\)comprehensive analysis showing model scale matters more than language\-specific training, while exposing persistent cultural understanding challenges\. VIVID provides a foundation for advancing figurative language understanding and developing culturally aware NLP systems\.

## 2\. Related Work

### 2\.1\. Idiom and Proverb Datasets

Existing figurative language datasets focus primarily on high\-resource languages\.Chakrabartyet al\.\([2022](https://arxiv.org/html/2608.03095#bib.bib21)\)introduce FLUTE, a synthetic dataset of 1000 English idiomatic sentences paired with synonymous or contradictory statements\.Tayyar Madabushiet al\.\([2022](https://arxiv.org/html/2608.03095#bib.bib22)\)present SemEval\-2022 Task 2 for multilingual idiomaticity detection across 8000 sentences in English, Portuguese, and Galician\.Azimeet al\.\([2025](https://arxiv.org/html/2608.03095#bib.bib24)\)compile ProverbEval with cultural proverbs from four Ethiopian languages and English, whileWanget al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib25)\)present KorID, a cloze\-test dataset of 1631 Korean idioms\. These datasets advance figurative language resources but lack coverage of Vietnamese, leaving a gap in evaluating culturally grounded expressions in this language\.

### 2\.2\. Vietnamese Benchmarks

Vietnamese NLP has established benchmarks for natural language inferenceHuynhet al\.\([2022](https://arxiv.org/html/2608.03095#bib.bib40)\), sentiment analysisHoet al\.\([2019](https://arxiv.org/html/2608.03095#bib.bib41)\), hate speech detectionHoanget al\.\([2023](https://arxiv.org/html/2608.03095#bib.bib36)\), and comprehensive NLU evaluation through ViGLUETranet al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib30)\)and VLUETranet al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib30)\)\. VIMQALeet al\.\([2022b](https://arxiv.org/html/2608.03095#bib.bib34)\)evaluates multi\-hop reasoning capabilities\. However, no dataset exists for assessing figurative and cultural language understanding in Vietnamese idioms and proverbs\. VIVID addresses this gap as the first benchmark for evaluating culturally grounded figurative expressions in Vietnamese\.

### 2\.3\. Figurative Language Evaluation

Evaluating figurative language understanding presents unique challenges, as traditional automatic metrics like BLEU and ROUGE fail to capture semantic nuances in idiomatic interpretationTanget al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib27)\); Wanget al\.\([2025a](https://arxiv.org/html/2608.03095#bib.bib44)\)\. Recent studies demonstrate that LLM\-as\-a\-Judge approachesZhenget al\.\([2023](https://arxiv.org/html/2608.03095#bib.bib38)\)achieve higher correlation with human judgment for figurative language tasks\.S\. Rezaeimanesh \([2025](https://arxiv.org/html/2608.03095#bib.bib11)\)shows strong alignment for Persian\-English idiom translation, whileTanget al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib27)\)andWanget al\.\([2025b](https://arxiv.org/html/2608.03095#bib.bib26)\),validate similar effectiveness for East Asian and Chinese idioms\. However, existing work primarily establishes that LLM judges outperform automatic metrics without systematically investigating how prompting strategies affect evaluation reliability for figurative language\. Different approaches—zero\-shot scoring, demonstration examples, chain\-of\-thought reasoning, or aspect\-based evaluation—may yield varying alignment with human judgment, yet this remains underexplored for culturally grounded expressions\. We address this gap through systematic comparison of four prompting strategies for Vietnamese idiom evaluation, establishing that aspect\-based evaluation achieves strongest human alignment \(Cohen’s κ = 0\.792\)\. This provides methodological guidance for reliable figurative language assessment in low\-resource contexts\.

## 3\. Dataset

![Refer to caption](https://arxiv.org/html/2608.03095v1/pipeline.drawio.png)Figure 1:Overview of the data collection and annotation pipeline\.We present VIVID, a culturally grounded evaluation benchmark dataset contains 1636 idioms and proverbs across 7 topics that capture Vietnamese cultural contexts, ranging from everyday life to specific subject areas, as well as Vietnamese grammar and linguistics

We further annotate the dataset into five distinct characteristics that LLMs are highly prone to misunderstanding\. Because idioms may exhibit multiple characteristics simultaneously, the total frequency counts exceed the number of idioms\. Dataset statistics are reported in Figure[2](https://arxiv.org/html/2608.03095#S3.F2)and[3](https://arxiv.org/html/2608.03095#S3.F3)\. The dataset is constructed in three stages:[3\.1](https://arxiv.org/html/2608.03095#S3.SS1)\. Data Collection ,[3\.2](https://arxiv.org/html/2608.03095#S3.SS2)\. Data Categorization and[3\.3](https://arxiv.org/html/2608.03095#S3.SS3)\. Data Validation, summarized in Figure[1](https://arxiv.org/html/2608.03095#S3.F1)

![Refer to caption](https://arxiv.org/html/2608.03095v1/barplot.png)Figure 2:Statistics of VIVID per characteristics\.![Refer to caption](https://arxiv.org/html/2608.03095v1/semantic_distribution.png)Figure 3:Statistics of VIVID per topics\.### 3\.1\. Data Collection

To construct VIVID, we adopted a systematic approach to gather high\-quality Vietnamese idioms and proverbs from authoritative lexicographic sources\. Our data collection process prioritized linguistic authenticity, cultural representativeness, and semantic diversity\.

We compiled idioms and proverbs from two complementary authoritative dictionaries\. The primary source was the dictionary by Nguyen LanLân \([2010](https://arxiv.org/html/2608.03095#bib.bib17)\), which provides a structured and comprehensive collection of widely\-used Vietnamese idioms and proverbs accompanied by clear definitions and semantic explanations\. This resource is recognized as a standard reference in Vietnamese linguistics and language education, ensuring the reliability and acceptance of the selected expressions

To enhance the linguistic diversity and capture regional variations, we incorporated additional entries from the dictionary compiled by Vu Dung and collaboratorsDung and others \([2000](https://arxiv.org/html/2608.03095#bib.bib19)\)\. Unlike Nguyen Lan’s work, which predominantly features standardized forms, the Vu Dung dictionary documents numerous idiomatic variants including regional expressions, alternative phrasings, and lexical variations that reflect the richness of Vietnamese idiomatic usage across different dialects and contexts\. This dual\-source approach enables our dataset to capture both standardized expressions and their naturally occurring variants in Vietnamese communication\.

Through this compilation process, we assembled a total of 2,254 idiom\-explanation pairs that form the foundation of the VIVID benchmark\.

![Refer to caption](https://arxiv.org/html/2608.03095v1/taxonomy.png)Figure 4:Taxonomy of idiomatic expressions that pose challenges for LLMs due to literal meaning, archaic vocabulary, cultural specificity, and other linguistic features\.
### 3\.2\. Data Categorization

#### 3\.2\.1\. Linguistic Complexity Taxonomy

Existing Vietnamese figurative language resources lack systematic categorization with respect to the linguistic and cultural features that challenge computational interpretation\. To address this gap, we developed a theory\-driven taxonomy grounded in consultation with Vietnamese language experts and informed by preliminary error analysis of LLM outputs\.

Our taxonomy organizes Vietnamese idioms and proverbs along three principal aspects and five distinct traits that capture the primary sources of interpretation difficulty, as illustrated in Figure[4](https://arxiv.org/html/2608.03095#S3.F4)

The Meaning Aspectencompasses idioms where semantic interpretation requires understanding beyond literal compositional meaning:

- •Literal expressions: Proverbs that are purely descriptive without metaphorical layers, yet LLMs frequently over\-interpret them by inferring non\-existent figurative meanings \(e\.g\., “Da đồng lông móc" describes physical characteristics of buffalo skin but is often misinterpreted as a metaphor for human personality\)
- •Pragmatic nuances: Expressions containing sarcasm, irony, or evaluative connotations that deviate from neutral interpretations \(e\.g\., “Tốt mẽ khoe màu" is intended to mock superficiality but is often misunderstood as promoting humility\)

The Lexical Aspectcaptures vocabulary\-related challenges arising from archaic or specialized terminology:

- •Sino\-Vietnamese terms: Expressions containing rare or literary Sino\-Vietnamese vocabulary infrequently encountered in contemporary usage \(e\.g\., “Ác khuất non đoài"\)
- •Uncommon vocabulary: Idioms featuring archaic or obsolete words no longer common in modern Vietnamese \(e\.g\., “Am thanh cảnh vắng," where “am" refers to a small rural temple\)

The Agricultural and Customary Knowledge Aspectincludes expressions rooted in traditional Vietnamese cultural practices:

Folk knowledge\-based expressions: Proverbs reflecting agricultural practices, seasonal cycles, or customary wisdom from traditional Vietnamese life \(e\.g\., “Cấy tháng Chạp đạp không đổ," a saying based on historical farming experience\)

Since idioms may exhibit multiple characteristics simultaneously, entries can be annotated with multiple traits\. The distribution of these complexity characteristics across the dataset is presented in Figure[2](https://arxiv.org/html/2608.03095#S3.F2), showing that pragmatic nuances \(46\.9%\) and uncommon vocabulary \(46\.1%\) are the most prevalent challenges, followed by Sino\-Vietnamese terms \(40\.2%\), folk knowledge\-based expressions \(18\.9%\), and literal expressions \(16\.5%\)\.

#### 3\.2\.2\. Semantic Theme Classification

To facilitate domain\-specific evaluation and capture the cultural dimensions of Vietnamese idioms, we categorized each entry according to seven semantic themes that reflect prevalent topics in Vietnamese daily life, folk wisdom, and moral education\. This classification schema was derived through thematic analysis of idiomatic content commonly encountered in colloquial usage, folk literature, and educational materials\.

The seven themes are defined as follows:

- •Criticism: Expressions conveying disapproval, mockery, or critical judgment of behaviors, traits, or situations
- •Life Lessons: Proverbs offering philosophical reflections, moral guidance, or practical wisdom about life
- •Work and Nature: Idioms related to labor, agricultural practices, natural phenomena, or traditional livelihoods
- •Other: Expressions addressing miscellaneous topics including abstract concepts, specific scenarios, or descriptions
- •Social Relationships: Idioms reflecting interpersonal dynamics within society, including family ties, friendships, neighborhood interactions, and communal relationships
- •Love: Expressions conveying romantic relationships, emotional bonds, or matters of the heart
- •Virtues: Idioms emphasizing moral standards, ethical values, or ideal personal qualities

Figure[3](https://arxiv.org/html/2608.03095#S3.F3)presents the distribution of idioms across these semantic categories, with Criticism \(28\.8%\) being the most represented, followed by Life Lessons \(24\.4%\), Work and Nature \(16\.6%\), Other \(10\.5%\), Social Relationships \(9\.5%\), Love \(7\.0%\), and Virtues \(3\.1%\)\.

### 3\.3\. Data Validation

To ensure the reliability and cultural appropriateness of our annotations, we implemented a rigorous two\-stage validation process combining automatic annotation with expert human validation\.Stage 1: Automatic Annotation\.We employed GPT\-4\.1 to automatically annotate each idiom\-explanation pair with one or more labels corresponding to the predefined linguistic complexity traits in our taxonomy \(Section 3\.2\.1\)\. This automated labeling enabled systematic identification of idioms that are particularly prone to misinterpretation due to semantic complexity or cultural specificity, thereby establishing a foundation for subsequent human validation\.Stage 2: Human Validation\.Two native Vietnamese speakers with expertise in Vietnamese linguistics independently reviewed all annotations through a three\-step process:

1. 1\.Initial Review: Each annotator examined the LLM\-generated complexity trait labels for contextual relevance and accuracy\. Entries with clearly inaccurate or culturally inappropriate labels were flagged for removal, while uncertain cases were marked for collaborative discussion\.
2. 2\.Refinement: For the semantic theme classification \(Section 3\.2\.2\), both annotators independently assigned each idiom\-explanation pair to one of the seven predefined themes based on its core semantic content\. Cases with disagreement or ambiguity were jointly reviewed, and both annotators collaboratively revised the labels to ensure alignment with the predefined taxonomy through consensus\-based adjudication\.
3. 3\.Final Curation: All disagreements were resolved through discussion between the two annotators\. Only idiom\-explanation pairs achieving unanimous agreement on both complexity traits and semantic themes were included in the final dataset\.

This validation process resulted in a high\-quality curated dataset comprising 1,636 idiom\-explanation pairs with validated annotations, providing a reliable foundation for evaluating LLMs’ understanding of culturally grounded Vietnamese figurative language\.

## 4\. Evaluation Framework and Experiment Setup

In this section, we introduce our figurative language evaluation framework, which integrates both*generative*and discriminative evaluations of Vietnamese idioms and proverbs under the LLM\-as\-a\-Judge paradigm\. This setup not only ensures reliable benchmarking but also extends prior evaluation research through prompting strategy analysis, role\-based instructions, and human–LLM judge alignment studies\.

### 4\.1\. Generative Evaluation

To evaluate generative capability, LLMs are asked to explain Vietnamese idioms and proverbs from VIVID\. We adopt an open\-ended question format rather than constrained multiple\-choice, as prior work has shown that in multiple\-choice settings, models may rely on superficial patterns rather than genuine understandingGriotet al\.\([2025](https://arxiv.org/html/2608.03095#bib.bib45)\)\. The quality of each generated explanation is scored using an LLM\-as\-a\-Judge frameworkGuet al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib5)\), with GPT\-4\.1 serving as an independent evaluator\. We test four prompting strategies during evaluation:\(1\) Zero\-shot, where the model is given only the explanation and asked to score it;\(2\) DemonstrationBrownet al\.\([2020](https://arxiv.org/html/2608.03095#bib.bib7)\), where several annotated examples of high\- and low\-quality explanations are provided beforehand;\(3\) Chain\-of\-Thought \(CoT\)Weiet al\.\([2022](https://arxiv.org/html/2608.03095#bib.bib1)\), in which the model is prompted to reason step\-by\-step before producing a score; and\(4\) Aspect\-based Evaluation\(see Figure[7](https://arxiv.org/html/2608.03095#A1.F7)\), where the model is instructed to explicitly consider multiple criteria such as factual accuracy, completeness and alignment with common native speaker interpretations\.

Each explanation is rated on a scale from 0 to 5 based on its correctness, informativeness, and cultural appropriateness\. These scores are then aggregated to compare performance across models and prompting strategies\. Prior studies have shown that automatic metrics such as BLEU or COMET fail to capture the nuances of figurative languageTanget al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib27)\); S\. Rezaeimanesh \([2025](https://arxiv.org/html/2608.03095#bib.bib11)\); Wanget al\.\([2025b](https://arxiv.org/html/2608.03095#bib.bib26)\)\. In contrast, GPT\-based evaluations demonstrate high correlation with human judgments, particularly in figurative interpretation tasksYanget al\.\([2025](https://arxiv.org/html/2608.03095#bib.bib28)\)\.

PromptMeanCohen’s KappaHuman Baseline3\.471\.000Zero\-shot3\.400\.774CoT3\.700\.786Demonstration3\.440\.783Aspect\-based3\.490\.792Table 1:Evaluation of different LLM\-as\-a\-judge prompting strategies compared to human\.#### 4\.1\.1\. LLM Judge Reliability Analysis

To examine the reliability of the LLM\-as\-a\-Judge approach, we conduct a controlled experiment on 200 Vietnamese idioms and proverbs randomly sampled from our dataset, VIVID\. For each entry, GPT\-4o generates an explanation under a Zero\-shot setting\. These outputs are then independently evaluated by GPT\-4\.1, using each of the four prompting strategies outlined above\. Each explanation is rated on three dimensions: semantic accuracy, interpretive completeness, and alignment with native speaker understanding, using a 0–5 scale\.

We collect human annotations following the procedure ofS\. Rezaeimanesh \([2025](https://arxiv.org/html/2608.03095#bib.bib11)\)for comparison with LLM judge\. Two native Vietnamese speakers independently annotate 150 idiomatic expressions each, with a 50\-sample overlap\. Human scores are assigned on the same scale \(0–5\), achieving strong inter\-human annotator reliability\.

As reported in Table[1](https://arxiv.org/html/2608.03095#S4.T1), Aspect\-based Evaluation achieves the strongest alignment with human \(κ=0\.792\\kappa=0\.792\), yielding the closest mean rating \(3\.49 to 3\.47\)\. Consequently, we adapt Aspect\-based as the default prompting strategy for automatic scoring of figurative language generative task\.

### 4\.2\. Discriminative Evaluation

We also conduct multiple\-choice classification tasks focusing on \(1\) topic identification and \(2\) linguistic characteristic classification, using the prompts illustrated in Figure[9](https://arxiv.org/html/2608.03095#A2.F9)and Figure[10](https://arxiv.org/html/2608.03095#A2.F10)\. These tasks are implemented using the Language Model Evaluation Harness frameworkGaoet al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib29)\)\. To ensure a consistent and reproducible evaluation protocol across both opened and closed source models, we adopt exact\-match scoring\. This choice is motivated by the current limitation of certain closed models \(e\.g\., GPT and Gemini\) that do not expose log\-likelihoods\. All models are evaluated under a 3\-shot prompt to ensure understanding of task format and output consistency\.

### 4\.3\. Benchmark Models

We evaluate a diverse set of open\-source and closed\-source large language models \(LLMs\) that vary in scale, language specialization, and accessibility\. The open\-source models include Llama\-4\-Scout, Llama\-SEA\-LION\-v3\-8B\-IT, GreenMind\-Medium\-14B\-R1, Vistral\-7B\-Chat, VinaLLaMA\-7B\-Chat, and Qwen\-3\-14B, while the closed\-source models include GPT\-4o and Gemini 2\.5 Flash \(see Table[2](https://arxiv.org/html/2608.03095#S4.T2)for details\)\.

TypeModelAPI\-only LLMsGPT\-4o, Gemini 2\.5 FlashOpen\-source LLMsMultilingual:Llama\-4\-Scout \(109B\)Llama\-SEA\-LION\-v3\-8B\-IT \(8B\)Qwen\-3 \(14B\)Vietnamese\-specialized:GreenMind\-Medium\-14B\-R1 \(14B\)Tunget al\.\([2025](https://arxiv.org/html/2608.03095#bib.bib9)\)Vistral\-7B\-Chat \(7B\)Vo \([2024](https://arxiv.org/html/2608.03095#bib.bib8)\)VinaLLaMA\-7B \(7B\)Nguyenet al\.\([2023](https://arxiv.org/html/2608.03095#bib.bib6)\)Table 2:Model selection\. The numbers in parentheses indicate the parameters for models\.Two prompting strategies are employed: Zero\-shot \(Figure[5](https://arxiv.org/html/2608.03095#A1.F5)\) and Few\-shot promptingLiet al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib10)\)\. For Few\-shot prompting \(Figure[6](https://arxiv.org/html/2608.03095#A1.F6)\), three examples are provided per task, consisting of two proverbs and one idiom, selected to maximize coverage of the linguistic and cultural characteristics discussed above\. This design helps models better infer the task format and expected outputs\.

## 5\. Result & Discussion

### 5\.1\. Performance on Generative Task

CategoryModelPromptLoveVirtuesCriticismWork and natureSocietyLife LessonsOthersAverageOpen\-sourceLLMsVistral\-7B\-ChatZero\-shot0\.250\.490\.240\.340\.420\.440\.390\.35Few\-shot0\.541\.370\.580\.690\.690\.830\.700\.70GreenMind\-Medium\-14B\-R1Zero\-shot1\.051\.400\.990\.901\.041\.371\.061\.09Few\-shot0\.951\.120\.940\.910\.861\.280\.971\.02Vinallama\-7b\-chatZero\-shot0\.130\.160\.080\.110\.100\.220\.100\.13Few\-shot0\.230\.330\.140\.160\.310\.410\.170\.24Llama\-4\-Scout\-17B\-16EZero\-shot0\.861\.270\.920\.760\.880\.961\.010\.91Few\-shot1\.041\.611\.171\.191\.241\.391\.191\.24Llama\-SEA\-LION\-v3\-8B\-ITZero\-shot0\.460\.800\.540\.540\.500\.720\.550\.58Few\-shot0\.620\.610\.600\.680\.560\.780\.590\.65Qwen\-3\-14BZero\-shot1\.151\.750\.951\.011\.071\.280\.951\.09Few\-shot0\.861\.691\.011\.071\.071\.370\.991\.12API\-onlyLLMsGemini Flash 2\.5Zero\-shot2\.512\.942\.221\.762\.362\.692\.032\.29Few\-shot2\.542\.862\.512\.442\.522\.892\.462\.6GPT\-4oZero\-shot2\.333\.082\.322\.412\.582\.602\.442\.46Few\-shot1\.752\.901\.951\.911\.892\.311\.972\.04

Table 3:Mean score of the models by category \(rows: model and prompt style; columns: topic and average\)\. The highest score for each category is in bold\. The top\-performing open\-source models are marked in blue\.CategoryModelPromptOnly literal expressionsSino\-Vietnamese termsUncommon vocabularyFolk knowledge\-based expressionsPragmatic nuancesOpen\-sourceLLMsVistral\-7B\-ChatZero\-shot0\.400\.380\.270\.340\.24Few\-shot0\.760\.820\.570\.740\.58GreenMind\-Medium\-14B\-R1Zero\-shot0\.911\.300\.911\.040\.97Few\-shot0\.951\.200\.801\.060\.94Vinallama\-7b\-chatZero\-shot0\.070\.120\.060\.150\.10Few\-shot0\.120\.280\.180\.260\.18Llama\-4\-Scout\-17B\-16EZero\-shot0\.941\.180\.750\.830\.83Few\-shot1\.191\.501\.101\.251\.18Llama\-SEA\-LION\-v3\-8B\-ITZero\-shot0\.480\.740\.460\.580\.54Few\-shot0\.590\.790\.520\.700\.58Qwen\-3\-14BZero\-shot0\.851\.370\.921\.101\.00Few\-shot0\.961\.340\.921\.161\.03API\-onlyLLMsGemini Flash 2\.5Zero\-shot1\.703\.042\.302\.152\.19Few\-shot2\.343\.112\.452\.572\.55GPT\-4oZero\-shot2\.432\.862\.332\.392\.44Few\-shot1\.972\.361\.892\.041\.92

Table 4:Model performance across five types of linguistic and cultural complexity\.We evaluate eight LLMs on the idiom/proverb explanation generation task in VIVID, under Zero\-shot and Few\-shot prompting across seven semantic themes \(Table[3](https://arxiv.org/html/2608.03095#S5.T3)\)\. Smaller models \(7\-8B parameters\) struggle significantly, with VinaLLaMA\-7B\-Chat achieving only 0\.13 and Vistral\-7B\-Chat reaching 0\.35\. Medium\-sized models \(14B parameters\) perform better but remain below 1\.5, with both Vietnamese\-specialized GreenMind\-14B \(1\.09\) and multilingual Qwen\-3\-14B \(1\.09\) showing comparable performance\. This suggests that current language\-specific training does not provide clear advantages for cultural figurative understanding at similar model scales\. Large\-scale models demonstrate substantially higher scores, with GPT\-4o \(2\.46\) and Gemini Flash 2\.5 \(2\.29\) outperforming all open\-source systems, likely due to their extensive pretraining and significantly larger parameter counts\.

Nevertheless, all models struggle with idioms/proverbs \(best approach with GPT\-4o and zero\-shot prompting achieves less than 50% correctness on average\), underscoring persistent challenges in Vietnamese figurative understanding\.

#### 5\.1\.1\. Robustness To Linguistic Complexity

We analyze model robustness across five annotated traits: \(1\) literal expressions, \(2\) uncommon vocabulary, \(3\) rare Sino\-Vietnamese terms, \(4\) folk knowledge–based expressions, and \(5\) pragmatic nuances\.

Table[4](https://arxiv.org/html/2608.03095#S5.T4)presents the average performance per category, with the Sino\-Vietnamese terms category as the least challenging one for LLMs\. GPT\-4o and Gemini Flash 2\.5 outperform all others across these dimensions\. GPT\-4o achieves 2\.86 on Sino\-Vietnamese terms and 2\.44 on pragmatic nuances \(Zero\-shot\), while Gemini Flash 2\.5 \(Few\-shot\) reaches 2\.57 on folk knowledge and 2\.45 on rare vocabulary\. Conversely, lighter models such as VinaLLaMA\-7B and Vistral\-7B remain below 1\.0, highlighting their limited semantic and cultural generalization capacity\.

Few\-shot prompting generally improves performance across most categories, confirming the benefit of in\-context examples for linguistically complex content\. The largest gain is seen in Llama\-4\-Scout\-17B \(1\.18→1\.501\.18\\rightarrow 1\.50on Sino\-Vietnamese terms\)\. Notably, GPT\-4o shows the opposite trend: its scores drop slightly in Few\-shot, suggesting overfitting or semantic rigidity when exposed to structured exemplars\. We further analyze this anomaly in Section[5\.1\.2](https://arxiv.org/html/2608.03095#S5.SS1.SSS2)\.

Manual inspection reveals systematic failures: literal over\-interpretation \(imposing figurative readings where none exist\), lexical gaps \(mistranslating archaic vocabulary\), cultural disconnection \(losing contextual meaning in folk\-based idioms\), and pragmatic flattening \(replacing sarcasm or irony with neutral tone\)\. These patterns expose fundamental gaps in current models’ cultural competency\.

#### 5\.1\.2\. Investigating Prompting Failures in GPT\-4o on VIVID

GPT\-4o uniquely shows consistent performance decline from zero\-shot to few\-shot prompting\. Analysis reveals GPT\-4o overfits to exemplar patterns, imitating moralizing tone rather than focusing on semantic fidelity\. For example, “Chưa lại người" \(physical recovery after illness\) was correctly interpreted in zero\-shot but reinterpreted in few\-shot as metaphor for immaturity\. This indicates didactic examples induce stylistic mimicry and semantic drift, fabricating moral lessons absent in source idioms\.

Few\-shot prompting also degrades vocabulary handling\. “Lang đuôi thì bán, lang trán thì cày" \(about buffalo coat patterns\) was correctly interpreted under zero\-shot but misread under few\-shot, where “lang" was mistaken for “sweet potato," producing an unrelated agricultural metaphor\. Similar findings byPhelpset al\.\([2024](https://arxiv.org/html/2608.03095#bib.bib20)\)suggest few\-shot prompting can hinder figurative interpretation by encouraging stylistic imitation over contextual understanding\.

### 5\.2\. Evaluating Topic and Linguistic Characteristic Classification Tasks

CategoryModelTopicLinguistics and culturalOpen\-source LLMsLlama\-SEA\-LION\-v3\-8B\-IT0\.3190\.013GreenMind\-Medium\-14B\-R10\.4550\.035Qwen\-3\-14B0\.4470\.031Llama\-4\-Scout\-17B\-16E0\.4590\.042Vinallama\-7B\-Chat0\.0790\.016Vistral\-7B\-Chat0\.0000\.000API\-based LLMsGPT\-4o0\.5240\.184Gemini Flash 2\.50\.5350\.144

Table 5:Model accuracy across topic and linguistic\-cultural classification tasks\.We evaluate model performance on two discriminative tasks: topic classification and linguistic–cultural taxonomy classification\. Accuracy results are summarized in Table[5](https://arxiv.org/html/2608.03095#S5.T5)\.

Open\-source LLMs show considerable variation across both tasks, while API\-based models consistently outperform them, likely due to larger parameter scales and broader pretraining\. Among open\-weight systems, Llama\-4\-Scout\-17B\-16E achieves the highest scores in both topic and linguistic–cultural labeling \(0\.459 and 0\.042\), followed by Qwen3\-14B \(0\.447, 0\.031\) and GreenMind\-Medium\-14B\-R1 \(0\.455, 0\.035\)\. Nevertheless, accuracy on linguistic characteristic classification remains low overall, partly because the exact\-match metric penalizes even minor deviations \(e\.g\., added or omitted words\)\. For instance, Viet\-Mistral\-7B\-Chat scored zero in both tasks due to inconsistent adherence to output format and limited capacity \(7B parameters\)\. Closed\-source models such as GPT\-4o and Gemini 2\.5 Flash perform best, achieving 0\.524 and 0\.535 on topic classification, and 0\.184 and 0\.144 on linguistic–cultural labelling, respectively\.

Performance on linguistic–cultural characteristic classification is much lower than on topic classification, reflecting the complexity of culturally grounded expressions\. Errors often arise from difficulty interpreting idioms’ implicit semantics\. For example, “ăn tái ăn tam” \(“to eat rare, to eat three times”\), literally describing repetitive eating, belongs to the*Other*category but was misclassified by both GPT\-4o and Gemini 2\.5 Flash as*Criticism*, likely due to reliance on affective cues\. Similarly, “bánh giầy nếp cái, con gái họ” \(“glutinous rice cake, their daughter”\), a proverb metaphorically praising a beautiful girl \(*Virtues*\), was misclassified by GPT\-4o as*Love*and by Gemini 2\.5 Flash as Other\. These cases illustrate the persistent gap between surface\-level lexical cues and the culturally embedded semantics required for accurate classification\.

These results demonstrate that state\-of\-the\-art LLMs still struggle to fully capture the meaning and cultural dimensions of Vietnamese idioms and proverbs, underscoring the value of VIVID as a benchmark for advancing culturally aware NLP evaluation\.

## 6\. Conclusion

We present VIVID, the first systematic benchmark for evaluating culturally grounded figurative language understanding in Vietnamese, comprising 1,636 idioms and proverbs annotated with linguistic complexity traits and semantic themes\. Our extensive evaluation reveals that even state\-of\-the\-art models achieve less than 50% correctness on average, underscoring fundamental gaps in cultural language understanding\. Our analysis reveals systematic failure modes including literal over\-interpretation, lexical gaps with archaic vocabulary, cultural disconnection in folk\-based expressions, and pragmatic flattening\. Notably, few\-shot prompting does not universally improve performance, with GPT\-4o exhibiting stylistic overfitting that degrades semantic fidelity\. These findings demonstrate that current LLMs lack the cultural competency required for nuanced figurative interpretation despite their general linguistic capabilities\. VIVID establishes a robust evaluation framework for Vietnamese figurative language comprehension and provides actionable insights for developing culturally aware NLP systems that go beyond surface\-level pattern matching to achieve genuine figurative language understanding\.

## 7\. Limitations

First, our automatic evaluation framework relies on a proprietary closed\-source model \(GPT\-4\.1\)\. Dependence on a closed model raises concerns regarding transparency and long\-term reproducibility\. Commercial models may be updated or modified over time without full disclosure of technical details, potentially leading to changes in evaluation behavior in future versions\. This may affect the stability of results if experiments are replicated at different points in time\. Second, the reliability analysis between the LLM judge and human annotators is conducted on a subset of 200 samples due to the substantial cost and effort associated with manual annotation\. While this sample size is adequate for estimating agreement trends between evaluators, it may not fully capture the complete diversity of linguistic forms, rarity levels, and cultural nuances present in VIVID\. Future work may extend validation to a larger scale and compare multiple evaluation models, including open\-source alternatives, to further enhance the transparency and reproducibility of the proposed evaluation framework\.

## 8\. Acknowledgements

This publication has emanated from research supported in part by grants from Research Ireland under Grant \[12\-RC\-2289\-P2\] and \[18/CRT/6223\] which is co\-funded under the European Regional Development Fund\. For the purpose of Open Access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission\.

## 9\. Bibliographical References

- I\. A\. Azime, A\. L\. Tonja, T\. D\. Belay, Y\. Chanie, B\. F\. Balcha, N\. H\. Abadi, H\. B\. Ademtew, M\. A\. Nerea, D\. D\. Yadeta, D\. D\. Geremew, A\. A\. Tesfu, P\. Slusallek, T\. Solorio, and D\. Klakow \(2025\)ProverbEval: exploring LLM evaluation challenges for low\-resource language understanding\.InFindings of the Association for Computational Linguistics: NAACL 2025,Albuquerque, New Mexico,pp\. 6250–6266\.Cited by:[§1](https://arxiv.org/html/2608.03095#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.03095#S2.SS1.p1.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Brown, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)Cited by:[§4\.1](https://arxiv.org/html/2608.03095#S4.SS1.p1.1)\.
- FLUTE: figurative language understanding through textual explanations\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Abu Dhabi, United Arab Emirates,pp\. 7139–7159\.Cited by:[§1](https://arxiv.org/html/2608.03095#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.03095#S2.SS1.p1.1)\.
- V\. Dunget al\.\(2000\)Từ điển thành ngữ & tục ngữ việt nam \[dictionary of vietnamese idioms & proverbs\]\.Nhà xuất bản Văn Hóa Thông Tin,Hà Nội\(Vietnamese\)\.Cited by:[§3\.1](https://arxiv.org/html/2608.03095#S3.SS1.p3.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[Appendix B](https://arxiv.org/html/2608.03095#A2.p1.1),[§1](https://arxiv.org/html/2608.03095#S1.p4.1),[§4\.2](https://arxiv.org/html/2608.03095#S4.SS2.p1.1)\.
- M\. Griot, J\. Vanderdonckt, D\. Yuksel, and C\. Hemptinne \(2025\)Pattern recognition or medical knowledge? the problem with multiple‐choice questions in medicine\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 5321–5341\.External Links:[Link](https://aclanthology.org/2025.acl-long.266/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.266)Cited by:[§4\.1](https://arxiv.org/html/2608.03095#S4.SS1.p1.1)\.
- J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu, S\. Wang, K\. Zhang, Y\. Wang, W\. Gao, L\. Ni, and J\. Guo \(2024\)A survey on llm\-as\-a\-judge\.arXiv preprint arXiv:2411\.15594\.External Links:[Link](https://arxiv.org/abs/2411.15594)Cited by:[§4\.1](https://arxiv.org/html/2608.03095#S4.SS1.p1.1)\.
- V\. A\. Ho, D\. Nguyen, D\. H\. T\. Nguyen, L\. T\. Pham, D\. Nguyen, K\. V\. Nguyen, and N\. L\. Nguyen \(2019\)Emotion recognition for vietnamese social media text\.ArXivabs/1911\.09339\.External Links:[Link](https://api.semanticscholar.org/CorpusID:208202333)Cited by:[§1](https://arxiv.org/html/2608.03095#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.03095#S2.SS2.p1.1)\.
- P\. G\. Hoang, C\. D\. Luu, K\. Q\. Tran, K\. V\. Nguyen, and N\. L\. Nguyen \(2023\)ViHOS: hate speech spans detection for Vietnamese\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,A\. Vlachos and I\. Augenstein \(Eds\.\),Dubrovnik, Croatia,pp\. 652–669\.External Links:[Link](https://aclanthology.org/2023.eacl-main.47/),[Document](https://dx.doi.org/10.18653/v1/2023.eacl-main.47)Cited by:[§2\.2](https://arxiv.org/html/2608.03095#S2.SS2.p1.1)\.
- T\. V\. Huynh, K\. V\. Nguyen, and N\. L\. Nguyen \(2022\)ViNLI: a vietnamese corpus for studies on open\-domain natural language inference\.InInternational Conference on Computational Linguistics,External Links:[Link](https://api.semanticscholar.org/CorpusID:252819372)Cited by:[§1](https://arxiv.org/html/2608.03095#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.03095#S2.SS2.p1.1)\.
- N\. Lân \(2010\)Từ điển thành ngữ, tục ngữ việt nam \[dictionary of vietnamese idioms and proverbs\]\.Nhà xuất bản Văn Hóa Thông Tin,Hà Nội\(Vietnamese\)\.Cited by:[§3\.1](https://arxiv.org/html/2608.03095#S3.SS1.p2.1)\.
- K\. Le, H\. Nguyen, T\. Le Thanh, and M\. Nguyen \(2022a\)VIMQA: a Vietnamese dataset for advanced reasoning and explainable multi\-hop question answering\.InProceedings of the Thirteenth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 6521–6529\.External Links:[Link](https://aclanthology.org/2022.lrec-1.700/)Cited by:[§1](https://arxiv.org/html/2608.03095#S1.p1.1)\.
- K\. Le, H\. Nguyen, T\. L\. Thanh, and M\. Nguyen \(2022b\)VIMQA: a Vietnamese dataset for advanced reasoning and explainable multi\-hop question answering\.InProceedings of the Thirteenth Language Resources and Evaluation Conference,Marseille, France,pp\. 6521–6529\.External Links:[Link](https://aclanthology.org/2022.lrec-1.700)Cited by:[§2\.2](https://arxiv.org/html/2608.03095#S2.SS2.p1.1)\.
- D\. Li, B\. Jiang, L\. Huang, A\. Beigi, C\. Zhao, Z\. Tan, A\. Bhattacharjee, Y\. Jiang, C\. Chen, T\. Wu, K\. Shu, L\. Cheng, and H\. Liu \(2024\)From generation to judgment: opportunities and challenges of llm\-as\-a\-judge\.arXiv preprint arXiv:2411\.16594\.External Links:[Link](https://arxiv.org/abs/2411.16594)Cited by:[§4\.3](https://arxiv.org/html/2608.03095#S4.SS3.p2.1)\.
- E\. Liu, C\. Cui, K\. Zheng, and G\. Neubig \(2022\)Testing the ability of language models to interpret figurative language\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,M\. Carpuat, M\. de Marneffe, and I\. V\. Meza Ruiz \(Eds\.\),Seattle, United States,pp\. 4437–4452\.External Links:[Link](https://aclanthology.org/2022.naacl-main.330/),[Document](https://dx.doi.org/10.18653/v1/2022.naacl-main.330)Cited by:[§1](https://arxiv.org/html/2608.03095#S1.p1.1)\.
- Q\. Nguyen, H\. Pham, and D\. Dao \(2023\)VinaLLaMA: llama\-based vietnamese foundation model\.Technical reportTechnical ReportarXiv:2312\.11011,arXiv\.External Links:[Link](https://arxiv.org/abs/2312.11011)Cited by:[Table 2](https://arxiv.org/html/2608.03095#S4.T2.1.3.2.1.7)\.
- H\. Phê \(2021\)Từ điển tiếng việt \[vietnamese dictionary\]\.Nhà xuất bản Hồng Đức,Hà Nội\(Vietnamese\)\.Cited by:[§1](https://arxiv.org/html/2608.03095#S1.p1.1)\.
- D\. Phelps, T\. Pickard, M\. Mi, E\. Gow\-Smith, and A\. Villavicencio \(2024\)Sign of the times: evaluating the use of large language models for idiomaticity detection\.InProceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies \(MWE\-UD\) @ LREC\-COLING 2024,Torino, Italia,pp\. 178–187\.External Links:[Link](https://aclanthology.org/2024.mwe-1.22/)Cited by:[§5\.1\.2](https://arxiv.org/html/2608.03095#S5.SS1.SSS2.p2.1)\.
- Y\. Y\. S\. Rezaeimanesh \(2025\)Large language models for persian ←→ english idiom translation\.InProceedings of the 2025 North American Chapter of the Association for Computational Linguistics \(NAACL\),External Links:[Link](https://aclanthology.org/2025.naacl-long.405.pdf)Cited by:[§2\.3](https://arxiv.org/html/2608.03095#S2.SS3.p1.1),[§4\.1\.1](https://arxiv.org/html/2608.03095#S4.SS1.SSS1.p2.1),[§4\.1](https://arxiv.org/html/2608.03095#S4.SS1.p2.1)\.
- K\. Tang, P\. Song, Y\. Qin, and X\. Yan \(2024\)Creative and context\-aware translation of east Asian idioms with GPT\-4\.InProceedings of the Eleventh Workshop on Asian Translation \(WAT 2024\),Miami, Florida, USA,pp\. 1–21\.External Links:[Link](https://aclanthology.org/2024.wat-1.1/)Cited by:[§1](https://arxiv.org/html/2608.03095#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.03095#S2.SS3.p1.1),[§4\.1](https://arxiv.org/html/2608.03095#S4.SS1.p2.1)\.
- H\. Tayyar Madabushi, E\. Gow\-Smith, M\. Garcia, C\. Scarton, M\. Idiart, and A\. Villavicencio \(2022\)SemEval\-2022 task 2: multilingual idiomaticity detection and sentence embedding\.InProceedings of the 16th International Workshop on Semantic Evaluation \(SemEval\-2022\),Seattle, United States,pp\. 107–121\.External Links:[Link](https://aclanthology.org/2022.semeval-1.14)Cited by:[§2\.1](https://arxiv.org/html/2608.03095#S2.SS1.p1.1)\.
- M\. Tran, P\. Nguyen, L\. Nguyen, and D\. Dinh \(2024\)ViGLUE: a Vietnamese general language understanding benchmark and analysis of Vietnamese language models\.InFindings of the Association for Computational Linguistics: NAACL 2024,Mexico City, Mexico,pp\. 4174–4189\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.261)Cited by:[§2\.2](https://arxiv.org/html/2608.03095#S2.SS2.p1.1)\.
- L\. Q\. Tung, H\. Q\. Viet, P\. B\. Loc, and V\. T\. Thu \(2025\)GreenMind: a next\-generation vietnamese large language model for structured and logical reasoning\.Technical reportTechnical ReportarXiv:2504\.16832,arXiv\.External Links:[Link](https://arxiv.org/abs/2504.16832)Cited by:[Table 2](https://arxiv.org/html/2608.03095#S4.T2.1.3.2.1.5)\.
- J\. Vo \(2024\)Vi\-mistral\-x: building a vietnamese language model with advanced continual pre\-training\.Technical reportTechnical ReportarXiv:2403\.15470,arXiv\.External Links:[Link](https://arxiv.org/abs/2403.15470)Cited by:[Table 2](https://arxiv.org/html/2608.03095#S4.T2.1.3.2.1.6)\.
- M\. Wang, V\. T\. Pham, F\. Moghimifar, and T\. Vu \(2025a\)Proverbs run in pairs: evaluating proverb translation capability of large language model\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 1646–1662\.External Links:[Link](https://aclanthology.org/2025.findings-acl.83/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.83),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2608.03095#S1.p2.1),[§2\.3](https://arxiv.org/html/2608.03095#S2.SS3.p1.1)\.
- M\. Wang, V\. Pham, F\. Moghimifar, and T\. Vu \(2025b\)Proverbs run in pairs: evaluating proverb translation capability of large language model\.InFindings of the Association for Computational Linguistics: ACL 2025,External Links:[Link](https://aclanthology.org/2025.findings-acl.83)Cited by:[§2\.3](https://arxiv.org/html/2608.03095#S2.SS3.p1.1),[§4\.1](https://arxiv.org/html/2608.03095#S4.SS1.p2.1)\.
- X\. Wang, J\. Yeo, J\. Lim, and H\. Kim \(2024\)KULTURE bench: a benchmark for assessing language model in Korean cultural context\.InProceedings of the 38th Pacific Asia Conference on Language, Information and Computation,Tokyo, Japan,pp\. 914–927\.Cited by:[§1](https://arxiv.org/html/2608.03095#S1.p2.1),[§2\.1](https://arxiv.org/html/2608.03095#S2.SS1.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://arxiv.org/abs/2201.11903)Cited by:[§4\.1](https://arxiv.org/html/2608.03095#S4.SS1.p1.1)\.
- C\. Yang, Y\. Dou, D\. Heineman, X\. Wu, and W\. Xu \(2025\)Evaluating llms on chinese idiom translation\.InConference on Language Modeling \(COLM\),External Links:[Link](https://doi.org/10.48550/arXiv.2508.10421),[Document](https://dx.doi.org/10.48550/arXiv.2508.10421),2508\.10421Cited by:[§4\.1](https://arxiv.org/html/2608.03095#S4.SS1.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.ArXivabs/2306\.05685\.Cited by:[§2\.3](https://arxiv.org/html/2608.03095#S2.SS3.p1.1)\.

## Appendix APrompts

### A\.1\. Generative Task Prompts

We present the exact prompt templates used for explanation generation\. All prompts are issued in Vietnamese to ensure the models respond in the target language and leverage their Vietnamese\-language capabilities\.

##### Zero\-shot Prompt\.

The zero\-shot prompt \(Figure[5](https://arxiv.org/html/2608.03095#A1.F5)\) is intentionally minimal, providing only the idiom or proverb and asking for a concise Vietnamese explanation\.

VietnameseGiải thích ngắn gọn bằng tiếng Việtý nghĩa câu thành ngữ sau:\{phrase\}

English \(translation\)Briefly explain in Vietnamesethe meaning of the following idiom/proverb:\{phrase\}

Figure 5:Zero\-shot prompt for idiom explanation generation \(Vietnamese original left, English translation right\)\.
##### Few\-shot Prompt\.

The few\-shot prompt \(Figure[6](https://arxiv.org/html/2608.03095#A1.F6)\) situates the model as a Vietnamese language expert and provides three demonstrative examples covering diverse idiom types \(a social proverb, a culinary proverb, and an ironic proverb\) before asking for the target explanation\.

VietnameseBạn là chuyên gia ngôn ngữ tiếng Việt\. Nhiệm vụ của bạn làgiải thích bằng tiếng Việtý nghĩa của câu thành ngữ hoặc tục ngữ sau một cách ngắn gọn, rõ ràng và dễ hiểu\. Hãy đọc và tham khảo các cách giải thích dưới đây:Ví dụ 1:“dân chỉ công bộc”Giải thích: Câu tục ngữ “dân chỉ công bộc” có ý nghĩa là dân chúng chính là người chủ, còn những người làm việc trong bộ máy nhà nước, chính quyền là “công bộc”, tức là người phục vụ cho dân\. Thành ngữ này nhấn mạnh vai trò quan trọng của người dân trong xã hội và trách nhiệm của những người làm công tác quản lý, lãnh đạo trong việc phục vụ và đáp ứng nhu cầu của nhân dân\.Ví dụ 2:“Cần tái cải nhừ”Giải thích: Một kinh nghiệm ăn uống: với rau cần khi thả vào nước sôi, đun cho sôi lại là bắc ra ngay \(tái\) để rau không bị nhũn và tránh có mùi nồng; với rau cải phải đun kỹ mới hết hăng hết cay\.Ví dụ 3:“ăn cho thủng nổi trôi rế”Giải thích: diễn tả việc ăn uống một cách rất nhiều, không kiêng khem hay tiết chế\. Ý nghĩa của câu này là chỉ sự ăn uống quá mức, đến mức có thể “thủng” cả cái rế \(vật dụng đựng thức ăn\)\. Chê những kẻ ăn uống tham lam quá độ\.Bây giờ, hãy giải thích câu sau:\{phrase\}giải thích:

English \(translation\)You are a Vietnamese language expert\. Your task is toexplain in Vietnamesethe meaning of the following idiom or proverb concisely, clearly, and accessibly\. Please read and refer to the example explanations below:Example 1:“dân chỉ công bộc”Explanation: The proverb “dân chỉ công bộc” means that the people are the true masters, while those working in government and state apparatus are “công bộc” — servants of the people\. It emphasises the vital role of citizens in society and the responsibility of officials to serve and meet the needs of the populace\.Example 2:“Cần tái cải nhừ”Explanation: A culinary tip: water celery \(cần\) should be removed from boiling water as soon as it returns to a boil \(tái\) to stay crisp and avoid a pungent smell; mustard greens \(cải\) must be cooked thoroughly to lose their bitterness and sharpness\.Example 3:“ăn cho thủng nổi trôi rế”Explanation: Depicts eating in extreme excess without any restraint\. The saying criticises gluttonous behaviour so severe it could “pierce through” the rế \(a vessel used to hold food\)\. It mocks those who are insatiably greedy eaters\.Now, please explain the following idiom/proverb:\{phrase\}explanation:

Figure 6:Few\-shot prompt for idiom explanation generation, with three in\-context demonstrations \(Vietnamese original left, English translation right\)\.

### A\.2\. LLM\-as\-a\-Judge Prompt

The aspect\-based judge is shown in Figure[7](https://arxiv.org/html/2608.03095#A1.F7)\. It instructs GPT\-4\.1 to evaluate a generated explanation across four explicit dimensions before producing a single overall similarity score on a 0–5 scale\. Crucially, a score of 0 on Criterion 1 \(core\-meaning accuracy\) forces an overall score of 0 regardless of other criteria, preventing superficially fluent but semantically incorrect responses from receiving inflated scores\.

Vietnamese\[Hướng dẫn\]Bạn là chuyên gia ngôn ngữ\. Nhiệm vụ của bạn là đánh giá độ tương đồng giữa giải thích gốc \(con người\) và giải thích của LLM cho một câu thành ngữ/tục ngữ, dựa trên các tiêu chí sau:1\.Độ chính xác của ý nghĩa cốt lõi \(0–5 điểm\):Giải thích của LLM nắm bắt đúng và chính xác ý nghĩa chính của câu thành ngữ/tục ngữ so với giải thích gốc? \(5: Rất chính xác, 0: Sai lệch hoàn toàn\)2\.Khả năng nắm bắt sắc thái, ngữ cảnh, ẩn ý \(0–5 điểm\):Giải thích của LLM có phản ánh được các sắc thái, ngữ cảnh sử dụng điển hình, hoặc ý nghĩa ẩn sâu của câu nói như giải thích gốc không? \(5: Rất tương đồng, 0: Không phản ánh được\)3\.Chất lượng diễn đạt, sự rõ ràng, lưu loát \(0–5 điểm\):Giải thích của LLM có rõ ràng, dễ hiểu, mạch lạc và tự nhiên không? \(5: Rất tốt, 0: Rất kém/khó hiểu\)4\.Độ đầy đủ \(0–5 điểm\):Giải thích của LLM có bao gồm các ý chính được đề cập trong giải thích gốc không? \(5: Rất đầy đủ, 0: Thiếu nhiều ý chính\)5\.Độ tương đồng tổng thể \(0–5 điểm\):Tổng hợp các đánh giá trên, mức độ tương đồng chung là bao nhiêu? \(5: Rất tương đồng, 0: Không tương đồng\)\[In\-context examples — see Figure[8](https://arxiv.org/html/2608.03095#A1.F8)\]\[Nhiệm vụ cần đánh giá\]Thành ngữ/Tục ngữ:\{phrase\}Giải thích của Con người:\{ground\_truth\}Giải thích của LLM:\{llm\_output\}Đánh giá chi tiết:1\. Độ chính xác của ý nghĩa cốt lõi:2\. Khả năng nắm bắt sắc thái, ngữ cảnh, ẩn ý:3\. Chất lượng diễn đạt, sự rõ ràng, lưu loát:4\. Độ đầy đủ:Nếu tiêu chí 1 được 0 điểm thì Độ tương đồng tổng thể được 0\.Độ tương đồng tổng thể: hãy suy nghĩ sau đó chỉ trả về duy nhất đánh giá của độ tương đồng tổng thể và không giải thích gì thêm

English \(translation\)\[Instructions\]You are a language expert\. Your task is to evaluate the similarity between the original \(human\-written\) explanation and the LLM\-generated explanation for a given Vietnamese idiom or proverb, based on the following criteria:1\.Accuracy of core meaning \(0–5 points\):Does the LLM explanation correctly and precisely capture the main meaning of the idiom/proverb compared to the original explanation? \(5: Very accurate, 0: Completely incorrect\)2\.Capture of nuance, context, and implicit meaning \(0–5 points\):Does the LLM explanation reflect the nuances, typical usage contexts, or deeper implied meanings of the expression as in the original explanation? \(5: Highly similar, 0: Does not reflect them at all\)3\.Expression quality, clarity, and fluency \(0–5 points\):Is the LLM explanation clear, easy to understand, coherent, and natural? \(5: Excellent, 0: Very poor/difficult to understand\)4\.Completeness \(0–5 points\):Does the LLM explanation include the key points mentioned in the original explanation? \(5: Very complete, 0: Missing many key points\)5\.Overall similarity \(0–5 points\):Synthesising the above assessments, what is the overall degree of similarity? \(5: Highly similar, 0: Not similar at all\)\[In\-context examples — see Figure[8](https://arxiv.org/html/2608.03095#A1.F8)\]\[Task to evaluate\]Idiom/Proverb:\{phrase\}Human explanation:\{ground\_truth\}LLM explanation:\{llm\_output\}Detailed evaluation:1\. Accuracy of core meaning:2\. Capture of nuance, context, and implicit meaning:3\. Expression quality, clarity, and fluency:4\. Completeness:If criterion 1 receives 0 points, the overall similarity score must also be 0\.Overall similarity: Think carefully, then return only the overall similarity score and provide no further explanation\.

Figure 7:Aspect\-based LLM\-as\-a\-Judge prompt \(Vietnamese original left, English translation right\)\. GPT\-4\.1 outputs only the integer overall similarity score \(0–5\)\.#### A\.2\.1\. Judge Prompt In\-Context Examples

Two annotated examples are embedded in the judge prompt \(see Figure[8](https://arxiv.org/html/2608.03095#A1.F8)\) to calibrate the scoring scale: one high\-quality response \(score 5\) and one incorrect response \(score 1\)\.

VietnameseVí dụ Minh họa 1Thành ngữ/Tục ngữ:Chưa đủ lông đủ cánhGiải thích nghĩa gốc: Còn non nớt, chưa đủ khả năng tự lập\.Giải nghĩa do LLM: Câu thành ngữ “Chưa đủ lông đủ cánh” được dùng để chỉ một người hoặc một sự vật, sự việc nào đó chưa phát triển hoàn thiện, chưa đủ trưởng thành hoặc chưa có đủ khả năng, kinh nghiệm để thực hiện một nhiệm vụ, công việc nào đó một cách độc lập\.Đánh giá chi tiết: 1\. Độ chính xác của ý nghĩa cốt lõi: 5/5 2\. Khả năng nắm bắt sắc thái, ngữ cảnh, ẩn ý: 5/5 3\. Chất lượng diễn đạt, sự rõ ràng, lưu loát: 5/5 4\. Độ đầy đủ: 5/5 Độ tương đồng tổng thể: 5/5Ví dụ Minh họa 2Thành ngữ/Tục ngữ:Kim chỉ có đầuGiải thích nghĩa gốc: Phải có trên dưới, đầu đuôi, trật tự văn phép\.Giải nghĩa do LLM: Câu thành ngữ “Kim chỉ có đầu” mang ý nghĩa rằng mọi việc đều có nguyên nhân và nguồn gốc, không có gì tự nhiên mà xảy ra\.Đánh giá chi tiết: 1\. Độ chính xác của ý nghĩa cốt lõi: 1/5 2\. Khả năng nắm bắt sắc thái, ngữ cảnh, ẩn ý: 1/5 3\. Chất lượng diễn đạt, sự rõ ràng, lưu loát: 3/5 4\. Độ đầy đủ: 1/5 Độ tương đồng tổng thể: 1/5

English \(translation\)Illustrative Example 1Idiom/Proverb:Chưa đủ lông đủ cánh\(“Not yet fully feathered”\)Human explanation: Still inexperienced; not yet capable of independence\.LLM explanation: The idiom “Chưa đủ lông đủ cánh” is used to describe a person or thing that has not yet fully developed, is not sufficiently mature, or lacks the ability and experience to perform a task or role independently\.Detailed scores: 1\. Accuracy of core meaning: 5/5 2\. Capture of nuance, context, and implicit meaning: 5/5 3\. Expression quality, clarity, and fluency: 5/5 4\. Completeness: 5/5 Overall similarity: 5/5Illustrative Example 2\(low\-quality response\)Idiom/Proverb:Kim chỉ có đầu\(“The needle and thread have a head”\)Human explanation: There must be a proper order — top and bottom, beginning and end, social hierarchy and etiquette\.LLM explanation: The idiom “Kim chỉ có đầu” means that everything has a cause and origin; nothing happens spontaneously\.Detailed scores: 1\. Accuracy of core meaning: 1/5 2\. Capture of nuance, context, and implicit meaning: 1/5 3\. Expression quality, clarity, and fluency: 3/5 4\. Completeness: 1/5 Overall similarity: 1/5

Figure 8:In\-context scoring examples embedded in the judge prompt to calibrate the 0–5 scale \(Vietnamese original left, English translation right\)\.

## Appendix BDiscriminative Task Prompts

The discriminative evaluation is implemented via the Language Model Evaluation Harness framework\(Gaoet al\.,[2024](https://arxiv.org/html/2608.03095#bib.bib29)\)using exact\-match scoring under 3\-shot prompting\. All prompts are in Vietnamese\.

### B\.1\. Topic Classification Prompt

The topic classification prompt is shown in Figure[9](https://arxiv.org/html/2608.03095#A2.F9)\. Models are asked to identify which of seven semantic themes best describes a given idiom or proverb, returning a single integer \(1–7\)\.

VietnameseBạn là chuyên gia ngôn ngữ\. Dưới đây là một câu thành ngữ/tục ngữ: “\{phrase\}”Nhiệm vụ của bạn là xác định câu này thuộc một trong các đặc điểm sau:1\.Tình yêu đôi lứa và hôn nhân Ví dụ:chẳng nên tình trước nghĩa sau, có con ta gả cho nhau thiệt gì2\.Ca ngợi phẩm chất và đạo đức con người Ví dụ:cân quắc anh hùng3\.Phê phán và châm biếm các thói quen xấu, hành vi tiêu cực Ví dụ:ai biết uốn câu cho vừa miệng cá4\.Kinh nghiệm sống về sản xuất, lao động và thiên nhiên Ví dụ:ác tắm thì ráo, sáo tắm thì mưa5\.Các mối quan hệ xã hội, gia đình Ví dụ:ắm con chồng hơn bồng cháu ngoại6\.Triết lý và bài học về cách sống Ví dụ:con ruồi đậu nặng đồng cân7\.Các chủ đề khác\(nếu không phù hợp với các mục trên\)Chỉ trả lời duy nhất một số \(1–7\)\. Không giải thích, không thêm ký tự nào khác\.

English \(translation\)You are a language expert\. Below is a Vietnamese idiom or proverb: “\{phrase\}”Your task is to identify which of the following categories best describes it:1\.Romantic love and marriage Example:chẳng nên tình trước nghĩa sau, có con ta gả cho nhau thiệt gì2\.Praising human virtues and moral character Example:cân quắc anh hùng3\.Criticising and satirising bad habits or negative behaviour Example:ai biết uốn câu cho vừa miệng cá4\.Life experience about production, labour, and nature Example:ác tắm thì ráo, sáo tắm thì mưa5\.Social and family relationships Example:ắm con chồng hơn bồng cháu ngoại6\.Philosophy and life lessons Example:con ruồi đậu nặng đồng cân7\.Other topics\(if none of the above apply\)Reply with a single number only \(1–7\)\. No explanation, no extra characters\.

Figure 9:Topic classification prompt for thetopic\_labelingdiscriminative task \(Vietnamese original left, English translation right\)\.
### B\.2\. Linguistic Complexity Classification Prompt

The linguistic complexity classification prompt is presented in Figure[10](https://arxiv.org/html/2608.03095#A2.F10)\. Models are asked to identify*all applicable*complexity traits for a given idiom or proverb, returning a comma\-separated list of integers in ascending order \(e\.g\.,1,3,5\)\. An empty string is returned when no trait applies\. Exact\-match scoring penalises any deviation from the gold label, including reordering or additional characters\.

VietnameseBạn là chuyên gia ngôn ngữ\. Dưới đây là một câu thành ngữ/tục ngữ: “\{phrase\}”Nhiệm vụ của bạn là xác định câu này thuộcnhững đặc điểm nàotrong danh sách dưới đây\. Kết quả trả về là danh sách số thứ tự đặc điểm, cách nhau bằng dấu phẩy \(ví dụ: “1,3,5”\)\. Nếu không thuộc đặc điểm nào, trả về chuỗi rỗng “”\.Danh sách đặc điểm:1\. Là câu tục ngữkhông mang nghĩa bóng\(chỉ mang nghĩa đen/kỹ thuật\)\. Ví dụ:Da đồng lông móc\(kinh nghiệm chọn trâu, không hàm ý sâu xa\)\.2\. Cótừ Hán Việtxuất hiện\. Ví dụ:Nhất tự vi sư, bán tự vi sư\.3\. Có từít phổ biến, ít dùngtrong ngôn ngữ hiện đại\. Ví dụ:Am thanh cảnh vắng\(từ “am” nghĩa là chùa nhỏ\)\.4\. Thể hiệnkinh nghiệm dân gian xưacủa người Việt Nam\. Ví dụ:Cấy thưa thừa thóc, cấy mau dốc bồ\.5\. Mang sắc tháimỉa mai, châm biếm hoặc tiêu cực\. Ví dụ:Tốt mẽ khoe màu\.Chỉ trả lời bằng các số đặc điểm, sắp xếp theo thứ tự tăng dần, cách nhau bằng dấu phẩy\. Không giải thích, không thêm ký tự nào khác ngoài danh sách số\.

English \(translation\)You are a language expert\. Below is a Vietnamese idiom or proverb: “\{phrase\}”Your task is to identifyall applicable characteristicsfrom the list below\. Return a comma\-separated list of characteristic numbers in ascending order \(e\.g\., “1,3,5”\)\. If none apply, return an empty string “”\.List of characteristics:1\. The proverbcarries no figurative meaning\(literal/technical meaning only\)\. Example:Da đồng lông móc\(buffalo selection criteria, no deeper implication\)\.2\. ContainsSino\-Vietnamese vocabulary\. Example:Nhất tự vi sư, bán tự vi sư\.3\. Contains wordsrarely usedin modern Vietnamese\. Example:Am thanh cảnh vắng\(“am” means a small rural temple\)\.4\. Reflectstraditional Vietnamese folk knowledge\. Example:Cấy thưa thừa thóc, cấy mau dốc bồ\.5\. Carries asarcastic, ironic, or negative tone\. Example:Tốt mẽ khoe màu\.Reply only with the characteristic numbers in ascending order, comma\-separated\. No explanation, no extra characters beyond the number list\.

Figure 10:Linguistic complexity classification prompt for thepattern\_labelingdiscriminative task \(Vietnamese original left, English translation right\)\. Models must predict all applicable traits; exact match is required for a correct prediction\.

## Appendix CModel and Hyperparameter Details

For generation, all models usetemperature=0\.7\(default\) andmax\_tokens=150\. For judging, GPT\-4\.1 usestemperature=0andmax\_tokens=10\.