TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understanding

# A Unified Benchmark for Farsi Dialogue Generation and Understanding
Source: [https://arxiv.org/html/2609.01810](https://arxiv.org/html/2609.01810)
Neda Jamshidi, Kamyar Zeinalipour, Fahimeh Akbari, Monica Bianchini, Marco Maggini, Marco Gori

###### Abstract

Farsi, spoken by more than 120 million people, lacks a comprehensive benchmark for dialogue generation and understanding\. We introduceTalkFa, a unified benchmark comprising three complementary datasets: \(1\)Wiki\-FaDial, 4\.2K Wikipedia\-grounded dialogues for knowledge\-grounded generation; \(2\)DailyDialog\-FA, 6\.6K dialogues annotated for dialogue acts and emotions; and \(3\)PlayDial\-FA, 2\.1K theatrical dialogues with sentiment labels\. While LLMs assist data construction, every dialogue undergoes multi\-stage review and revision by native Farsi speakers, and only the final human\-approved dialogues are released\. Experiments with sixLLaMAandMistralmodels show that LoRA substantially improves dialogue generation while requiring only 25–50% of the training data to recover over 90% of the final performance gains\. Across classification tasks,FaBERTachieves the best dialogue\-act performance,LoRA\-Mistral\-7Bperforms best on emotion recognition, andMistral\-24Bachieves the highest sentiment score\. Human evaluation and independent external validation demonstrate the reliability of the benchmark, while comparisons with GPT\-4\.1 as an LLM judge reveal that automatic metrics substantially overestimate dialogue quality\. Zero\-shot evaluation with frontier LLMs further shows that TalkFa remains a challenging benchmark\. We will release all datasets, annotation guidelines, code, and checkpoints\.

![Refer to caption](https://arxiv.org/html/2609.01810v1/image11.png)Figure 1:The sample dialogues from three sources in the TalkFa dataset; full dialogues are provided in Appendix[A](https://arxiv.org/html/2609.01810#A1)\.## 1Introduction

Progress in conversational AI has been driven by public dialogue benchmarks such asDailyDialog[Li et al\. \(2017\)](https://arxiv.org/html/2609.01810#bib.bib1),EmpatheticDialogues[Rashkin et al\. \(2018\)](https://arxiv.org/html/2609.01810#bib.bib31), andTopical\-Chat[Gopalakrishnan et al\. \(2023\)](https://arxiv.org/html/2609.01810#bib.bib30)\. In contrast,Farsi111Also called Persian; we use the terms interchangeably\., spoken by over 120 million people, lacks a unified benchmark for dialogue generation and understanding\. Existing resources mainly focus on speech, sentiment, or general NLU \(e\.g\.,ParsiNLU[Khashabi et al\. \(2021\)](https://arxiv.org/html/2609.01810#bib.bib33)\), leaving conversational modeling largely unexplored\. Although multilingual encoders help alleviate data scarcity[Conneau et al\. \(2020\)](https://arxiv.org/html/2609.01810#bib.bib5), they often fail to capture culturally grounded conversational behavior\. Recent work, such asNusaDialogue[Purwarianti et al\. \(2025\)](https://arxiv.org/html/2609.01810#bib.bib35), demonstrates the effectiveness of LLM\-assisted, human\-verified dataset construction, but no comparable benchmark exists for Farsi\. The lack of a*public multi\-task benchmark*limits reproducible evaluation of Farsi dialogue systems, motivating the following research question:

Problem StatementHow can we build a reliable, culturally adapted benchmark for evaluating Farsi dialogue generation and understanding across multiple dialogue tasks?

To answer this question, we investigate five research questions:

RQ1\.How much does LoRA improve knowledge\-grounded Farsi dialogue generation?

RQ2\.Which models perform best for dialogue\-act and emotion classification?

RQ3\.How well do the same models perform on theatrical sentiment classification?

RQ4\.How do zero\-shot and LoRA\-adapted LLMs compare across TalkFa tasks?

RQ5\.How does generation quality scale with training data?

Contributions\.We introduceTalkFa, the first unified benchmark for Farsi dialogue, comprisingWiki\-FaDial,DailyDialog\-FA, andPlayDial\-FA, covering knowledge\-grounded dialogue generation, dialogue\-act classification, emotion recognition, and sentiment analysis \(Figure[1](https://arxiv.org/html/2609.01810#S0.F1)\)\. We present a reproducible LLM\-assisted, human\-curated pipeline through native\-speaker review and revision\. We establish strong baselines using LoRA\-adaptedLLaMAandMistralmodels together with multilingual and Farsi\-specific encoders\. Finally, we release all datasets, annotation guidelines, code, LoRA adapters, and fine\-tuned checkpoints to support reproducible Farsi dialogue research\.

## 2Related Work

Dialogue Benchmarks\.English dialogue benchmarks such asDailyDialog\([Li et al\., 2017](https://arxiv.org/html/2609.01810#bib.bib1)\),EmpatheticDialogues\([Rashkin et al\., 2018](https://arxiv.org/html/2609.01810#bib.bib31)\),Topical\-Chat\([Gopalakrishnan et al\., 2023](https://arxiv.org/html/2609.01810#bib.bib30)\),Wizard of Wikipedia\([Dinan et al\., 2018](https://arxiv.org/html/2609.01810#bib.bib38)\),PersonaChat\([Zhang et al\., 2018](https://arxiv.org/html/2609.01810#bib.bib3)\), andMultiWOZ\([Budzianowski et al\., 2018](https://arxiv.org/html/2609.01810#bib.bib29)\)have established benchmarks for open\-domain, empathetic, knowledge\-grounded, and task\-oriented dialogue\. Beyond English, Chinese resources includeKdConv\([Zhou et al\., 2020](https://arxiv.org/html/2609.01810#bib.bib41)\)andDuConv\([Wu et al\., 2019](https://arxiv.org/html/2609.01810#bib.bib42)\), while multilingual benchmarks such asxDial\-Eval\([Zhang et al\., 2023](https://arxiv.org/html/2609.01810#bib.bib43)\),MEGA\([Ahuja et al\., 2023](https://arxiv.org/html/2609.01810#bib.bib44)\), andMTOP\([Li et al\., 2020](https://arxiv.org/html/2609.01810#bib.bib32)\)address multilingual dialogue evaluation, broader generative language model evaluation, and task\-oriented semantic parsing, respectively\. However, no existing benchmark jointly supports Farsi dialogue generation together with multiple dialogue understanding tasks\.

Knowledge\-Grounded Dialogue\.Knowledge\-grounded dialogue has received increasing attention through datasets such asWizard of Wikipedia\([Dinan et al\., 2018](https://arxiv.org/html/2609.01810#bib.bib38)\),FaithDial\([Dziri et al\., 2022a](https://arxiv.org/html/2609.01810#bib.bib39)\), andCMU\-DoG\([Zhou et al\., 2018](https://arxiv.org/html/2609.01810#bib.bib52)\), alongside evaluation benchmarks such asBEGIN\([Dziri et al\., 2022b](https://arxiv.org/html/2609.01810#bib.bib40)\)\. In Farsi,Wiki\-FaDialextends this direction using GPT\-4o\-assisted dialogue generation followed by native\-speaker post\-editing and automatic and human evaluation\.

Dialogue Evaluation\.Automatic metrics including BLEU\([Papineni et al\., 2002](https://arxiv.org/html/2609.01810#bib.bib47)\), ROUGE\([Lin, 2004](https://arxiv.org/html/2609.01810#bib.bib48)\), and BERTScore\([Zhang et al\., 2019](https://arxiv.org/html/2609.01810#bib.bib46)\)remain widely used despite their limitations for open\-ended dialogue\([Liu et al\., 2016](https://arxiv.org/html/2609.01810#bib.bib45)\)\. More recent work introduced learned evaluators such asUSR\([Mehri and Eskenazi, 2020b](https://arxiv.org/html/2609.01810#bib.bib49)\),FED\([Mehri and Eskenazi, 2020a](https://arxiv.org/html/2609.01810#bib.bib54)\), and LLM\-as\-a\-judge methods\([Zheng et al\., 2023](https://arxiv.org/html/2609.01810#bib.bib50)\)\. Consistent with these observations, our experiments show that automatic metrics substantially overestimate Farsi dialogue quality relative to human judgments\.

Synthetic Data Generation and Farsi Resources\.LLM\-assisted dialogue generation has become increasingly common for low\-resource languages\([Suresh et al\., 2025](https://arxiv.org/html/2609.01810#bib.bib37);[Lee et al\., 2023](https://arxiv.org/html/2609.01810#bib.bib51)\)\.NusaDialogue\([Purwarianti et al\., 2025](https://arxiv.org/html/2609.01810#bib.bib35)\)further demonstrates the use of LLM\-assisted generation combined with human validation for dialogue resources in underrepresented languages\. While multilingual pretrained models such as mT5\([Xue et al\., 2021](https://arxiv.org/html/2609.01810#bib.bib4)\)provide broad cross\-lingual capabilities, multilingual coverage alone does not ensure culturally grounded conversational modeling\. Resources such asArabCulture\([Sadallah et al\., 2025](https://arxiv.org/html/2609.01810#bib.bib34)\)emphasize the importance of culture\-aware language technologies\. Existing Farsi resources mainly focus on NLU or general LLM evaluation, includingParsiNLU\([Khashabi et al\., 2021](https://arxiv.org/html/2609.01810#bib.bib33)\)andFarsEval\-PKBETS\([Shamsfard et al\., 2025](https://arxiv.org/html/2609.01810#bib.bib36)\), whileFaBERT\([Shahid Beheshti University NLP Group, 2024](https://arxiv.org/html/2609.01810#bib.bib16)\)andParsBERT\([HooshvareLab, 2020a](https://arxiv.org/html/2609.01810#bib.bib15)\)are pretrained Farsi language models for downstream tasks\. In contrast, to the best of our knowledge,TalkFais the first unified Farsi benchmark supporting knowledge\-grounded dialogue generation, dialogue\-act classification, emotion recognition, and sentiment analysis\. Table[1](https://arxiv.org/html/2609.01810#S2.T1)comparesTalkFawith representative dialogue benchmarks and existing Farsi language resources\.

DatasetLanguageSizeGAESHDailyDialogEnglish13K✓✓✓✗✓Wizard of WikipediaEnglish22K✓✗✗✗✓MultiWOZEnglish10K✓✓✗✗✓KdConvChinese4\.5K✓✗✗✗✓DuConvChinese30K✓✗✗✗✓DiaSetArabic23\.2K✗✗✗✓✓NusaDialogueMultilingual30K✓✗✗✗✓MTOPMultilingual100K✗✗✗✗✓ParsiNLUFarsi14\.5K✗✗✗✗✓FarsEval\-PKBETSFarsi4K✗✗✗✗✓TalkFa\(ours\)Farsi12\.9K✓✓✓✓✓Table 1:Comparison with representative English and multilingual dialogue benchmarks together with existing Farsi language resources\. G: dialogue generation; A: dialogue\-act classification; E: emotion classification; S: sentiment classification; H: human annotation or validation\. Dataset sizes are reported in their original units and may not be directly comparable\.†FarsEval\-PKBETSis a general LLM evaluation benchmark rather than a dialogue dataset\.
## 3TalkFa Creation

Table 2:Corpus statistics for the three TalkFa datasets\. “Parag” denotes Wiki\-FaDial\-Parag, “Wiki” denotes Wiki\-FaDial, “Daily” denotes DailyDialog\-FA, and “Play” denotes PlayDial\-FA\.Built from three complementary sources,TalkFacovers knowledge\-grounded dialogue \(Wiki\-FaDial, 4\.2K six\-turn chats\), everyday conversation \(DailyDialog\-FA, 6\.6K dialogues with turn\-level act and emotion labels\), and dramatic dialogue \(PlayDial\-FA, 2\.1K dialogues with dialogue\-level sentiment\)\.

### 3\.1Wiki\-FaDial: Context\-Aware Casual Farsi Dialogues

We study*context\-aware casual conversation generation in Farsi*: given an expository paragraph, the goal is to generate a short, informal dialogue that remains faithful to the source while sounding natural and idiomatic in everyday Farsi\.Wiki\-FaDialaddresses this gap by pairing Farsi encyclopaedic paragraphs with multi\-turn conversations in a conversational register\.

##### Construction\.

Source discovery\.We collected pages from ten curated indices of pedagogically valuable Farsi Wikipedia content, including*Featured Articles*,*Vital Articles*, and*Most\-Viewed by Topic*[Wikipedia contributors \(2025b\)](https://arxiv.org/html/2609.01810#bib.bib23);[Wikipedia contributors \(2025a\)](https://arxiv.org/html/2609.01810#bib.bib24);[Wikipedia contributors \(2014\)](https://arxiv.org/html/2609.01810#bib.bib26);[Wikipedia contributors \(2009\)](https://arxiv.org/html/2609.01810#bib.bib25)\. This yielded8,894pages spanning mathematics, history, biology, literature, and other core disciplines\.

Content filtering\.We retained only the lead paragraph of each article and applied three filters: \(i\) alength filter, removing paragraphs with fewer than 100 words; \(ii\) apolicy filter, excluding sensitive or disallowed content; and \(iii\) atopicality filter, retaining self\-contained definitions, key facts, or short narratives\. This resulted in4,184high\-quality source paragraphs\.

Prompting and generation\.Each paragraph was converted into a six\-turn dialogue usingGPT\-4o; the full prompt is provided in Appendix[B](https://arxiv.org/html/2609.01810#A2)\. The prompt required informal, culturally appropriate Farsi, exactly six turns, 1–20 words per turn, and areferencefield in every turn to quote or summarize the source paragraph for traceability\. The model returned one JSON object per paragraph with the key"dialogue", whose value is an array of six \{speaker,text,reference\} triples\.

##### Human Verification\.

Two native Farsi annotators222Both annotators are authors of this paper and performed annotation voluntarily without financial compensation\.with prior NLP experience reviewed and revised every generated dialogue\. Both annotators were born and raised in Iran and are familiar with Farsi linguistic conventions and cultural context\. In the first stage, one annotator corrected grammatical errors, normalized slang, inserted appropriate discourse markers, and verified the factual consistency of allreferencesnippets against the source paragraph\. In the second stage, the other annotator independently reviewed fluency, internal consistency, and faithfulness to the source text\. Because every generated turn includes areferencefield, both annotators explicitly verified that each utterance remained grounded in the original paragraph and introduced no unsupported information\. Remaining disagreements were resolved through adjudication\. Detailed annotation guidelines and editing examples are provided in Appendix[C](https://arxiv.org/html/2609.01810#A3)and Appendix[D](https://arxiv.org/html/2609.01810#A4)\.

##### Dataset Statistics\.

Wiki\-FaDialfeatures\.Table[2](https://arxiv.org/html/2609.01810#S3.T2)summarizes corpus statistics for the source paragraphs \(Wiki\-FaDial\-Parag\) and the generated dialogues \(Wiki\-FaDial\)\. The dataset contains4,184dialogues and25,104turns, with a median of81tokens per dialogue \(mean81\.79\)333Using the[bert\-fa](https://huggingface.co/HooshvareLab/bert-fa-base-uncased)tokenizer\.\. Compared withDailyDialog\-FAandPlayDial\-FA, it has similar dialogue length but higher lexical diversity \(TTR words0\.11; lemmas0\.09\) and relatively low length variance \(token std\.8\.46\) due to the fixed six\-turn design\.

Figure[2](https://arxiv.org/html/2609.01810#S3.F2)shows thatWiki\-FaDialdialogues cluster around 80–90 tokens, whereas the source paragraphs are substantially longer\. Figure[3](https://arxiv.org/html/2609.01810#S3.F3)shows balanced POS coverage across major grammatical categories, including nouns, verbs, adpositions, pronouns, auxiliaries, and conjunctions444We performed Farsi POS tagging with Hazm and fell back to Stanza if needed\.\.

Faithfulness without Copying\.To measure semantic grounding, we computed embedding similarity between each generated dialogue and its corresponding Wikipedia paragraph usingbert\-fasentence embeddings\. The average BLEU\-4 score is 0\.005 and ROUGE\-L F1is 0\.10, indicating minimal lexical overlap, while an average embedding cosine similarity of 0\.66 confirms that the generated dialogues remain semantically faithful to their source paragraphs\.

Figure 2:Dialogue token distributions across corpora\.Figure 3:POS counts \(bars\) and per\-unit means \(lines\) across corpora\.

### 3\.2DailyDialog\-FA: Everyday Farsi Dialogues for Turn\-Level Act & Emotion Classification

We study*turn\-level*dialogue\-act and emotion classification\. As a culturally localized counterpart to the EnglishDailyDialogcorpus,DailyDialog\-FAprovides Farsi supervision for both tasks while preserving the original annotation scheme\. Construction\.Label Inventory\.We preserve the originalDailyDialogtaxonomy, comprising four dialogue\-act categories \(*commissive*,*directive*,*inform*, and*question*\) and seven emotion categories \(*anger*,*disgust*,*fear*,*happiness*,*sadness*,*surprise*, and*no emotion*\)\.

Source Corpus and Translation\.We translate the EnglishDailyDialogcorpus[Li et al\. \(2017\)](https://arxiv.org/html/2609.01810#bib.bib1)into Farsi usingGPT\-4ounder constraints that enforce natural spoken Farsi, culturally appropriate references, preservation of dialogue flow, and alignment with Iranian politeness norms\. The complete translation prompt is provided in Appendix[E](https://arxiv.org/html/2609.01810#A5)\.

Human Verification\.Two native Farsi annotators reviewed and revised every translated dialogue in two stages\. In the first stage, one annotator improved fluency, localized cultural references \(e\.g\., currencies, places, and names\), corrected grammatical inconsistencies, and verified semantic fidelity to the English source\. In the second stage, the other annotator independently reviewed coherence, clarity, and translation accuracy\. Remaining issues were resolved through adjudication, and all dialogues were validated for formatting consistency and correct JSON structure\. Since annotators verified translations rather than assigning new labels, inter\-annotator agreement is not applicable\. Detailed guidelines and editing examples are provided in Appendix[F](https://arxiv.org/html/2609.01810#A6)and Appendix[G](https://arxiv.org/html/2609.01810#A7)\.

Dataset Statistics\.Table[2](https://arxiv.org/html/2609.01810#S3.T2)summarizes the corpus statistics\.DailyDialog\-FAcontains6,599dialogues,52,018turns, and635,942tokens\. Dialogues average7\.88turns and96\.37tokens \(median95; std\. dev\.26\.64\)\. The dataset exhibits the lexical characteristics of everyday conversational Farsi \(types13,624; lemmas10,664; TTR0\.03\)\. Figure[2](https://arxiv.org/html/2609.01810#S3.F2)shows that dialogue lengths are concentrated in the short\-to\-medium range, while Figure[3](https://arxiv.org/html/2609.01810#S3.F3)demonstrates balanced POS coverage across nouns, verbs, pronouns, auxiliaries, and connectives\.

### 3\.3PlayDial\-FA: Dramatic Farsi Dialogues for Dialogue\-Level Sentiment Classification

We study*dialogue\-level sentiment classification*in highly expressive dramatic conversations\. Compared with the everyday and knowledge\-grounded settings ofDailyDialog\-FAandWiki\-FaDial, theatrical dialogue exhibits stronger emotional expression, richer rhetorical language, and greater stylistic variation\.

Construction\.Corpus Harvesting\.We collected 30 public\-domain Farsi plays \(1940–2020\) from theIranNLP Drama Archive[TakBook \(2025\)](https://arxiv.org/html/2609.01810#bib.bib27)and converted them from PDF to text using OCR[PDF2Go \(2025\)](https://arxiv.org/html/2609.01810#bib.bib28)\. One annotator removed stage directions, formatting artifacts, and OCR errors, while a second annotator verified the cleaned dialogues against the original scripts\.

Segmentation and Reformulation\.UsingGPT\-4o[OpenAI \(2024\)](https://arxiv.org/html/2609.01810#bib.bib57), the plays were segmented into multi\-turn dialogues based on*scene continuity*; the complete prompt is provided in Appendix[H](https://arxiv.org/html/2609.01810#A8)\. To avoid verbatim reproduction while preserving the original style, each dialogue was reformulated byGPT\-4oand subsequently reviewed and revised by both annotators for fluency, coherence, and fidelity to the source plays\. Detailed guidelines and editing examples are provided in Appendix[I](https://arxiv.org/html/2609.01810#A9)and Appendix[J](https://arxiv.org/html/2609.01810#A10)\.

Human Verification\.Sentiment Annotation\.Each dialogue was assigned one of three sentiment labels:*negative*,*neutral*, or*positive*\. Initial labels were generated byGPT\-4oand independently reviewed by both annotators using the full play context\. Disagreements were resolved through discussion until consensus was reached\. Inter\-annotator agreement reached95\.89%withCohen’sκ\\kappa= 0\.929, indicating near\-perfect annotation reliability\.

Dataset Statistics\.Table[2](https://arxiv.org/html/2609.01810#S3.T2)summarizes the corpus statistics\.PlayDial\-FAcontains2,070dialogues,16,499turns, and200,158tokens\. Dialogues average7\.97turns and96\.69tokens \(median91; std\. dev\.46\.53\)\. Compared withDailyDialog\-FA, the dataset exhibits higher lexical diversity \(TTR words0\.08; lemmas0\.07\) and greater length variability, reflecting the expressive nature of theatrical dialogue\. Figure[2](https://arxiv.org/html/2609.01810#S3.F2)shows a broader dialogue\-length distribution, while Figure[3](https://arxiv.org/html/2609.01810#S3.F3)demonstrates balanced POS coverage across verbs, nouns, pronouns, auxiliaries, connectives, and expressive modifiers relevant to sentiment classification\.

### 3\.4Human Revision Analysis and Dataset Quality

All three datasets underwent the human verification procedures described earlier\. To quantify human intervention, we computed normalized character\-level edit distances between the initial GPT\-4o outputs and the final human\-approved dialogues using Python’sSequenceMatcher\. Annotators were free to revise dialogue structure, wording, speaker turns, pragmatic expressions, and cultural references rather than merely correcting grammatical errors\.

Wiki\-FaDial\.All 4,184 GPT\-4o\-generated dialogues were reviewed by native Farsi speakers following Appendix[C](https://arxiv.org/html/2609.01810#A3)\. Overall, 2,661 dialogues \(63\.0%\) required manual revision before approval, indicating substantial human intervention\.

DailyDialog\-FA\.Of the 6,599 translated dialogues, 1,123 \(17\.0%\) required revision\. At the turn level, 1,485 of 52,018 turns \(2\.9%\) were modified, with a mean edit rate of 12\.7% \(median 9\.5%\)\. Most edits improved fluency, phrasing, or cultural adaptation rather than correcting translation errors\.

PlayDial\-FA\.The 2,070 reformulated dialogues underwent two review rounds\. In the first, 395 dialogues \(19\.1%\) were revised \(mean edit rate 8\.2%\); in the second, only 28 \(1\.4%\) required additional changes \(mean edit rate 10\.4%\), indicating convergence toward stable, human\-approved quality\. Table[3](https://arxiv.org/html/2609.01810#S3.T3)summarizes the post\-editing statistics\. The high proportion of manually revised dialogues, particularly inWiki\-FaDial, confirms that TalkFa is an LLM\-assisted, human\-curated benchmark\. All released dialogues were reviewed and approved by native Farsi speakers\.

Table 3:Human post\-editing statistics\. Edited is the percentage of modified items, and Edit Rate is the mean normalized character\-level edit distance over modified items only\.#### 3\.4\.1Independent External Validation

To assess dataset quality and annotation reproducibility, an independent native Farsi speaker evaluated 300 stratified random samples \(100 per annotation task\)\.Wiki\-FaDialachieved 2\.8/3\.00 factual grounding, 3\.65/4\.00 naturalness, and 2\.70/3\.00 cultural appropriateness\.DailyDialog\-FAachieved 3\.00/3\.00 meaning preservation, 3\.90/4\.00 naturalness, and 2\.8/3\.00 cultural localization, whilePlayDial\-FAobtained 3\.50/4\.00 for dialogue naturalness\. Agreement with the released annotations reached 85% \(κ=0\.800\\kappa=0\.800\) for dialogue acts, 89% \(κ=0\.872\\kappa=0\.872\) for emotions, and 87% \(κ=0\.805\\kappa=0\.805\) for sentiment, indicating substantial to almost perfect agreement\. Full details are provided in Appendix[L](https://arxiv.org/html/2609.01810#A12)\.

## 4Experiments

Table 4:Base vs\. \+LoRA performance onWiki\-FaDial\(test\)\. Bold marks the best score for each metric\.Table 5:Macro\-averaged Precision \| Recall \| F1forAct,Emotion, andSentiment, computed from the model outputs\. In each model–category pair, the higher F1is in bold\. Higher is better\.Table 6:macro\-averaged Precision \| Recall \| F1forAct,Emotion, andSentiment, under two fine\-tuning regimes \(MLP vs\. Full\)\. In each triplet the higher F1is in bold\.This section evaluatesTalkFaon two tasks:*dialogue generation*and*utterance\-level classification*\.

Model families\.We evaluate two model groups\.*\(A\) Instruction\-tuned LLMs*\(generation and zero\-shot classification\):Llama\-3\.2\-1B[Meta AI \(2024e\)](https://arxiv.org/html/2609.01810#bib.bib6),Llama\-3\.2\-3B[Meta AI \(2024f\)](https://arxiv.org/html/2609.01810#bib.bib7),Llama\-3\.1\-8B[Meta AI \(2024d\)](https://arxiv.org/html/2609.01810#bib.bib8),Mistral\-7B[Mistral AI \(2024a\)](https://arxiv.org/html/2609.01810#bib.bib9),Mistral\-Nemo\-12B[Mistral AI \(2024b\)](https://arxiv.org/html/2609.01810#bib.bib10), andMistral\-Small\-24B[Mistral AI \(2025\)](https://arxiv.org/html/2609.01810#bib.bib11), spanning 1B–24B parameters\.*\(B\) Sentence encoders*\(supervised classification\):Multilingual\-E5Small/Base/Large[Microsoft \(2024c\)](https://arxiv.org/html/2609.01810#bib.bib12);[Microsoft \(2024a\)](https://arxiv.org/html/2609.01810#bib.bib13);[Microsoft \(2024b\)](https://arxiv.org/html/2609.01810#bib.bib14),ParsBERT[HooshvareLab \(2020a\)](https://arxiv.org/html/2609.01810#bib.bib15),FaBERT[Shahid Beheshti University NLP Group \(2024\)](https://arxiv.org/html/2609.01810#bib.bib16), mBERT[Google \(2024\)](https://arxiv.org/html/2609.01810#bib.bib17), RoBERTa\-Base/Large[Meta AI \(2024a\)](https://arxiv.org/html/2609.01810#bib.bib18);[Meta AI \(2024b\)](https://arxiv.org/html/2609.01810#bib.bib19), and XLM\-R Base/Large[Meta AI \(2019\)](https://arxiv.org/html/2609.01810#bib.bib20);[Meta AI \(2024c\)](https://arxiv.org/html/2609.01810#bib.bib21)\. This setup enables comparison between multilingual and Farsi\-specific pretraining across model scales\.Training regimes\.LLMs are evaluated in zero\-shot and LoRA\-adapted settings\([Hu et al\., 2022](https://arxiv.org/html/2609.01810#bib.bib2)\)\. For generation, one adapter is trained onWiki\-FaDial; for classification, separate adapters are trained onDailyDialog\-FAandPlayDial\-FA\. Only Lora adapters and task heads are updated\. Encoders are evaluated under two regimes: \(i\) frozen backbone with an MLP classifier head, and \(ii\) full fine\-tuning of encoder and classifier jointly\.

Training details\.Experiments are conducted on 3×\\timesNVIDIA RTX A6000 GPUs \(48GB\) using bf16 precision, FlashAttention, and gradient checkpointing\.*LLM LoRA fine\-tuning:*3 epochs; AdamW; cosine scheduler; learning rate1×10−41\\times 10^\{\-4\}; weight decay1×10−41\\times 10^\{\-4\}; batch size 4 with gradient accumulation 2; LoRA rank 128,α\\alpha256, dropout 0\.1\.*Encoder classification:*Frozen\-backbone models use a 4\-layer MLP head \(512→\\rightarrow256→\\rightarrow128→\\rightarrow64\) with NLLLoss for 100 epochs \(batch size 64; LR1×10−51\\times 10^\{\-5\}\)\. Full fine\-tuning updates encoder and classifier jointly for 3 epochs \(batch size 32; LR2×10−52\\times 10^\{\-5\}\)\.

Evaluation\.For classification, we report accuracy, macro\-precision, macro\-recall, and macro\-F1\. For dialogue generation, we report BLEU\-4, ROUGE\-L, BERTScore\-F usingxlm\-roberta\-large, and BERTCos usingbert\-fa[HooshvareLab \(2020b\)](https://arxiv.org/html/2609.01810#bib.bib22)sentence embeddings\.

### 4\.1Wiki\-FaDial: Dialogue Generation \(RQ1\)

We evaluate six instruction\-tuned LLMs in bothBaseand\+LoRAsettings onWiki\-FaDial555All experiments use a 90/5/5 train/validation/test split\.using the six\-turn generation schema from Sec\.[3\.1](https://arxiv.org/html/2609.01810#S3.SS1)\. Overall performance\.LoRA consistently improves performance across all metrics and model sizes \(Table[4](https://arxiv.org/html/2609.01810#S4.T4)\)\. Gains are largest for smaller models:Llama\-3\.2\-1BandLlama\-3\.2\-3Bnearly double their ROUGE\-L scores after fine\-tuning, while larger models \(Mistral\-Nemo,Mistral\-Small\-24B\) show smaller but stable improvements\. Notably, a LoRA\-adapted 7B model rivals or exceeds a 24B Base model, suggesting that*adaptation matters more than scale*for this task\. However, automatic metrics provide only a partial picture: models achieving BERTCos scores above 0\.97 still obtain only moderate human ratings\.

#### 4\.1\.1Beyond Automatic Metrics: Human and LLM\-Based Evaluation

We complement automatic metrics with human evaluation, GPT\-4\.1\-based[OpenAI \(2025\)](https://arxiv.org/html/2609.01810#bib.bib56)judging, multilingual semantic scoring, and zero\-shot evaluation using frontier LLMs\. Human evaluation\.We evaluate 7,200 generated dialogues across all six LLM families, including both Base and \+LoRA variants\. Each annotator independently rates 3,600 dialogues on a five\-point scale from highly natural and coherent \(A=5\) to incoherent or irrelevant \(E=1\)\. One annotator additionally re\-annotates a shared subset of 1,200 dialogues for agreement estimation\. Detailed guidelines and examples are provided in Appendix[K](https://arxiv.org/html/2609.01810#A11)\. Table[7](https://arxiv.org/html/2609.01810#S4.T7)reports the results\.

Table 7:Human evaluation \(1–5\) onWiki\-FaDial\.The strongest model \(Mistral\-Small\-24B\+LoRA\) achieves only2\.74/5, indicating that the benchmark remains far from saturated\. Re\-annotation yields 84\.96% raw agreement and Cohen’sκ=0\.759\\kappa=0\.759[Landis and Koch \(1977\)](https://arxiv.org/html/2609.01810#bib.bib53), indicating substantial agreement\. Zero\-shot frontier LLMs \(RQ4\)\.To assess whether frontier commercial models trivially solve TalkFa tasks, we evaluate GPT\-4\.1, GPT\-4\.1\-Nano, and DeepSeek\-V4\-Flash[DeepSeek\-AI \(2026\)](https://arxiv.org/html/2609.01810#bib.bib55)in a zero\-shot setting \(Table[8](https://arxiv.org/html/2609.01810#S4.T8)\)\.

Table 8:Zero\-shot results on TalkFa\.GPT\-4\.1 achieves 73\.7% accuracy on dialogue acts and 66\.4% on sentiment classification, but emotion classification remains difficult for all models \(macro\-F1≤0\.35\\leq 0\.35\), partly due to strong class imbalance\. LLM\-as\-a\-judge\.We also use GPT\-4\.1 \(temperature = 0\) to rate the same 7,200 generated dialogues on the A–E scale\. Table[9](https://arxiv.org/html/2609.01810#S4.T9)compares GPT\-4\.1 ratings against human annotations\.

Table 9:GPT\-4\.1 judge vs\. human annotators\.ρ\\rho: Spearman \(p<0\.001p<0\.001\);±\\pm1: % within one rating;κlin\\kappa\_\{\\rm lin\}: weighted Cohen’sκ\\kappa\.GPT\-4\.1 shows moderate agreement with human preferences but is systematically more lenient, assigning on average \+0\.45 higher grades\. Data\-efficiency ablation \(RQ5\)\.LoRA fine\-tuning is repeated using\{25%,50%,75%,100%\}\\\{25\\%,50\\%,75\\%,100\\%\\\}ofWiki\-FaDial\. Performance improves rapidly with the first 25% of training data, and 25–50% already recovers over 90% of final gains\. Full results are provided in Appendix[M](https://arxiv.org/html/2609.01810#A13)\. The Automatic–Human Gap\.Automatic metrics substantially overestimate dialogue quality: models exceeding 0\.93 BERTScore and 0\.97 BERTCos still receive only moderate human ratings \(2\.74/5\)\. This suggests that reference\-based metrics alone are insufficient for evaluating open\-ended Farsi dialogue\. Detailed comparisons are provided in Appendix[N](https://arxiv.org/html/2609.01810#A14)\.

### 4\.2DailyDialog\-FA: Turn\-Level Act & Emotion Classification \(RQ2\)

We compare six instruction\-tuned decoder LMs \(Base vs\. \+LoRA\) and ten encoders \(MLP vs\. Full\) forDialogue\-ActandEmotionclassification \(Tables[5](https://arxiv.org/html/2609.01810#S4.T5)–[6](https://arxiv.org/html/2609.01810#S4.T6)\)\. Dialogue acts\.Farsi\-specific encoders perform best overall:FaBERTachieves the highest act macro\-F1\(0\.75\), outperforming larger multilingual models\. LoRA\-adapted LLMs narrow the gap—Mistral\-24Breaches 0\.72—but do not surpass smaller Farsi\-pretrained encoders, suggesting that language\-specific pretraining is especially beneficial for structured classification in morphologically rich languages such as Farsi\. Emotions\.Emotion classification remains the most difficult TalkFa task, with no model exceeding 0\.38 macro\-F1\. This likely reflects both severe class imbalance \(the dominant “no emotion” class\) and cultural mismatch between English emotion categories and Farsi pragmatic conventions such as*ta’arof*\. Interestingly, multilingual similarity models \(E5\-Base/Large\) slightly outperform Farsi\-specific encoders, suggesting that cross\-lingual transfer may help low\-frequency emotion categories\. Practical recommendation\.Adaptation is the main driver of performance: LoRA consistently improves decoder LMs, while full fine\-tuning outperforms frozen encoders\. Notably, LoRA\-adapted 7B models approach the performance of 12B–24B variants, makingMistral\-7Ba strong trade\-off between quality and efficiency\.

### 4\.3PlayDial\-FA: Sentiment Classification \(RQ3\)

We evaluate dialogue\-level sentiment classification \(negative / neutral / positive\) on expressive theatrical dialogues \(Tables[5](https://arxiv.org/html/2609.01810#S4.T5)–[6](https://arxiv.org/html/2609.01810#S4.T6)\)\. LoRA unlocks LLM potential\.Base LLMs struggle on literary dialogue, typically showing high precision but very low recall by overpredicting the majority class\. LoRA substantially improves performance:Mistral\-24Bachieves the best result \(0\.62 macro\-F1\) by recovering minority\-class recall\. Encoders\.Full fine\-tuning benefits some multilingual backbones \(E5\-Base: 0\.56;mBERT: 0\.54\) but harms others\. Notably, Farsi\-specific encoders \(FaBERT,ParsBERT\) drop to≈\\approx0\.21 macro\-F1under full fine\-tuning, while frozen\-backbone MLP settings achieve≈\\approx0\.49\. This suggests that, given the relatively small size ofPlayDial\-FA, full fine\-tuning overfits and degrades pretrained representations\. Remaining challenges\.Despite having only three classes, theatrical sentiment remains difficult due to sarcasm, rhetorical exaggeration, and subtle polarity distinctions\. The best model still reaches only 0\.62 macro\-F1, leaving substantial room for improvement\.

## 5Conclusion and Future Work

TalkFaintroduces the first unified benchmark for Farsi dialogue generation and understanding, combiningWiki\-FaDial,DailyDialog\-FA, andPlayDial\-FA\. Using LLM\-assisted data creation with native\-speaker verification, we establish reproducible baselines for generation, dialogue acts, emotions, and sentiment classification\. LoRA substantially improves generation \(\+13 ROUGE\-LF1F\_\{1\},∼\\sim39% BLEU\-4\) and consistently outperforms zero\-shot settings\.FaBERTachieves the best dialogue\-act score \(0\.75 macro\-F1\),LoRA\-Mistral\-7Bthe best emotion score \(0\.38\), and LoRA\-Mistral\-24Bthe best sentiment result \(0\.62\)\. Moreover, 25–50% of the training data recovers over 90% of final generation gains\. Human evaluation reveals that even the best fine\-tuned model \(Mistral\-Small\-24B\+LoRA\) achieves only 2\.74/5 despite very high automatic scores \(BERTCos = 0\.97; BERTScore = 0\.93\), highlighting the limitations of reference\-based metrics for open\-ended dialogue generation\. Frontier LLMs achieve competitive but non\-saturating zero\-shot performance, while GPT\-4\.1 as a judge shows moderate agreement with human ratings \(ρ=0\.43\\rho=0\.43,±\\pm1 = 79\.7%\) but remains systematically more lenient\. We release all datasets, annotation guidelines, code, LoRA adapters, checkpoints, and baselines\. Future work includes richer pragmatic annotations, retrieval\-grounded dialogue, speech and code\-switching extensions, adaptation to Dari/Tajik, and robustness and safety evaluation suites with a public leaderboard\.

## 6Limitations

WhileTalkFaadvances evaluation for Farsi dialogue generation and understanding, several limitations remain\.

- •LLM\-Assisted Data Construction and Potential Contamination\.TalkFa was constructed with LLM assistance, using GPT\-4o for dialogue generation, translation, segmentation, or reformulation depending on the dataset, followed by multi\-stage human verification and post\-editing by native speakers\. Although this process substantially improves fluency, factual consistency, and cultural appropriateness, subtle stylistic regularities or lexical preferences inherited from the underlying model may still persist, potentially underrepresenting the diversity of fully human\-authored conversations\. In addition, some source materials, such as Wikipedia passages and public\-domain theatrical scripts, may overlap with the pretraining corpora of contemporary LLMs\. Because such overlap cannot be reliably verified, we do not claim that TalkFa is contamination\-free\. Future work could further investigate these effects by comparing TalkFa with newly available human\-authored Farsi dialogue resources and by developing more rigorous contamination detection methods\.
- •Structural Constraints inWiki\-FaDial:Wiki\-FaDialuses a fixed six\-turn structure with short utterances \(1–20 tokens per turn\)\. While this improves consistency and controllability, it underrepresents longer, multi\-topic, overlapping, or multi\-party conversations common in real\-world dialogue\. Models trained on the corpus may therefore become biased toward concise and orderly interactions\.
- •Cross\-Lingual Label Transfer inDailyDialog\-FA:Dialogue\-act and emotion labels are inherited from the EnglishDailyDialogtaxonomy\. However, emotional expression and pragmatic intent in Farsi do not always align cleanly with English conversational categories\. Phenomena such as*ta’arof*, indirectness, politeness mitigation, and culturally dependent emotional expression may not be fully captured by the transferred label inventory\.
- •Synthetic Reformulation inPlayDial\-FA:AlthoughPlayDial\-FAoriginates from real theatrical scripts, dialogues are segmented and reformulated usingGPT\-4oto avoid verbatim reproduction and standardize formatting\. This process may alter stylistic nuances, rhetorical structure, or author\-specific writing patterns present in the original plays\.
- •Limited Register and Dialect Coverage:AlthoughTalkFaspans encyclopedic, everyday, and theatrical dialogue, it does not cover many important real\-world settings such as social media interactions, spoken disfluencies, code\-switching, online slang, or multi\-party chats\. The benchmark also focuses primarily on standard Iranian Farsi and does not systematically include regional or closely related varieties such as Dari or Tajik\.
- •Evaluation Limitations:Despite incorporating human evaluation and LLM\-as\-a\-judge analysis, dialogue evaluation remains inherently subjective\. Human ratings may vary across annotators and cultural backgrounds, while LLM judges can exhibit systematic biases such as verbosity preference or leniency\. In addition, automatic metrics \(e\.g\., BLEU, ROUGE, BERTScore\) correlate imperfectly with human judgments for open\-ended conversational generation\.
- •Benchmark Scale and Model Coverage:WhileTalkFais substantially larger than prior Farsi dialogue resources, it remains smaller than major English conversational benchmarks\. Similarly, our experiments cover representative open\-source LLMs and encoders but do not exhaustively evaluate all architectures, prompting strategies, retrieval\-based systems, or reasoning\-oriented models\.
- •Independent Validation Scope:Our external validation relied on a single independent annotator evaluating stratified samples of each task\. While the resulting agreement was consistently high, future work could extend this evaluation by involving multiple independent annotators and larger validation sets to further assess annotation reproducibility\.

## References

- Ahujaet al\.\(2023\)K\. Ahuja, H\. Diddee, R\. Hada, M\. Ochieng, K\. Ramesh, P\. Jain, A\. Nambi, T\. Ganu, S\. Segal, M\. Ahmed,et al\.Mega: multilingual evaluation of generative ai\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 4232–4267\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p1.1)\.
- Budzianowskiet al\.\(2018\)P\. Budzianowski, T\. Wen, B\. Tseng, I\. Casanueva, S\. Ultes, O\. Ramadan, and M\. GasicMultiwoz\-a large\-scale multi\-domain wizard\-of\-oz dataset for task\-oriented dialogue modelling\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 5016–5026\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p1.1)\.
- Conneauet al\.\(2020\)A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. StoyanovUnsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 8440–8451\.Cited by:[§1](https://arxiv.org/html/2609.01810#S1.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-v4\-flash\.Note:Hugging FaceModel cardExternal Links:[Link](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash)Cited by:[§4\.1\.1](https://arxiv.org/html/2609.01810#S4.SS1.SSS1.p2.1)\.
- Dinanet al\.\(2018\)E\. Dinan, S\. Roller, K\. Shuster, A\. Fan, M\. Auli, and J\. WestonWizard of wikipedia: knowledge\-powered conversational agents\.arXiv preprint arXiv:1811\.01241\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p1.1),[§2](https://arxiv.org/html/2609.01810#S2.p2.1)\.
- Dziriet al\.\(2022a\)N\. Dziri, E\. Kamalloo, S\. Milton, O\. Zaiane, M\. Yu, E\. M\. Ponti, and S\. ReddyFaithdial: a faithful benchmark for information\-seeking dialogue\.Transactions of the Association for Computational Linguistics10,pp\. 1473–1490\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p2.1)\.
- Dziriet al\.\(2022b\)N\. Dziri, H\. Rashkin, T\. Linzen, and D\. ReitterEvaluating attribution in dialogue systems: the begin benchmark\.Transactions of the Association for Computational Linguistics10,pp\. 1066–1083\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p2.1)\.
- Google \(2024\)GoogleGoogle\-bert/bert\-base\-multilingual\-cased\.Note:HuggingFace model card\. Multilingual BERT trained on 104 Wikipedia languagesExternal Links:[Link](https://huggingface.co/google-bert/bert-base-multilingual-cased)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Gopalakrishnanet al\.\(2023\)K\. Gopalakrishnan, B\. Hedayatnia, Q\. Chen, A\. Gottardi, S\. Kwatra, A\. Venkatesh, R\. Gabriel, and D\. Hakkani\-TurTopical\-chat: towards knowledge\-grounded open\-domain conversations\.arXiv preprint arXiv:2308\.11995\.Cited by:[§1](https://arxiv.org/html/2609.01810#S1.p1.1),[§2](https://arxiv.org/html/2609.01810#S2.p1.1)\.
- HooshvareLab \(2020a\)HooshvareLabHooshvareLab/bert\-base\-parsbert\-uncased\.Note:HuggingFace model card\. Monolingual Persian BERT model \(ParsBERT\)External Links:[Link](https://huggingface.co/HooshvareLab/bert-base-parsbert-uncased)Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p4.1),[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- HooshvareLab \(2020b\)HooshvareLabHooshvareLab/bert\-fa\-base\-uncased\.Note:HuggingFace model card\. Persian language model ParsBERT v2\.0External Links:[Link](https://huggingface.co/HooshvareLab/bert-fa-base-uncased)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p4.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Khashabiet al\.\(2021\)D\. Khashabi, A\. Cohan, S\. Shakeri, P\. Hosseini, P\. Pezeshkpour, M\. Alikhani, M\. Aminnaseri, M\. Bitaab, F\. Brahman, S\. Ghazarian,et al\.Parsinlu: a suite of language understanding challenges for persian\.Transactions of the Association for Computational Linguistics9,pp\. 1147–1162\.Cited by:[§1](https://arxiv.org/html/2609.01810#S1.p1.1),[§2](https://arxiv.org/html/2609.01810#S2.p4.1)\.
- Landis and Koch \(1977\)J\. R\. Landis and G\. G\. KochThe measurement of observer agreement for categorical data\.biometrics,pp\. 159–174\.Cited by:[Table 11](https://arxiv.org/html/2609.01810#A12.T11),[§4\.1\.1](https://arxiv.org/html/2609.01810#S4.SS1.SSS1.p2.1)\.
- Leeet al\.\(2023\)D\. Lee, J\. Pujara, M\. Sewak, R\. White, and S\. JauharMaking large language models better data creators\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 15349–15360\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p4.1)\.
- Liet al\.\(2020\)H\. Li, A\. Arora, S\. Chen, A\. Gupta, S\. Gupta, and Y\. MehdadMTOP: a comprehensive multilingual task\-oriented semantic parsing benchmark\.arXiv preprint arXiv:2008\.09335\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p1.1)\.
- Liet al\.\(2017\)Y\. Li, H\. Su, X\. Shen, W\. Li, Z\. Cao, and S\. NiuDailydialog: a manually labelled multi\-turn dialogue dataset\.InProceedings of the Eighth International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 986–995\.Cited by:[§1](https://arxiv.org/html/2609.01810#S1.p1.1),[§2](https://arxiv.org/html/2609.01810#S2.p1.1),[§3\.2](https://arxiv.org/html/2609.01810#S3.SS2.p2.1)\.
- Lin \(2004\)C\. LinRouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p3.1)\.
- Liuet al\.\(2016\)C\. Liu, R\. Lowe, I\. V\. Serban, M\. Noseworthy, L\. Charlin, and J\. PineauHow not to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 2122–2132\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p3.1)\.
- Mehri and Eskenazi \(2020a\)S\. Mehri and M\. EskenaziUnsupervised evaluation of interactive dialog with dialogpt\.InProceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue,pp\. 225–235\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p3.1)\.
- Mehri and Eskenazi \(2020b\)S\. Mehri and M\. EskenaziUSR: an unsupervised and reference free evaluation metric for dialog generation\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 681–707\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p3.1)\.
- Meta AI \(2019\)Meta AIFacebookAI/xlm\-roberta\-base\.Note:HuggingFace model card\. Multilingual RoBERTa pretrained on CommonCrawl \(100 languages\)External Links:[Link](https://huggingface.co/FacebookAI/xlm-roberta-base)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Meta AI \(2024a\)Meta AIFacebookAI/roberta\-base\.Note:HuggingFace model card\. Pre\-trained English RoBERTa\-base modelExternal Links:[Link](https://huggingface.co/FacebookAI/roberta-base)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Meta AI \(2024b\)Meta AIFacebookAI/roberta\-large\.Note:HuggingFace model card\. RoBERTa\-large model introduced in the RoBERTa paperExternal Links:[Link](https://huggingface.co/FacebookAI/roberta-large)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Meta AI \(2024c\)Meta AIFacebookAI/xlm\-roberta\-large\.Note:HuggingFace model card\. Large multilingual XLM\-RoBERTa modelExternal Links:[Link](https://huggingface.co/FacebookAI/xlm-roberta-large)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Meta AI \(2024d\)Meta AIMeta\-llama/llama\-3\.1\-8b\-instruct\.Note:HuggingFace model cardExternal Links:[Link](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Meta AI \(2024e\)Meta AIMeta\-llama/llama\-3\.2\-1b\-instruct\.Note:HuggingFace model cardExternal Links:[Link](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Meta AI \(2024f\)Meta AIMeta\-llama/llama\-3\.2\-3b\-instruct\.Note:HuggingFace model cardExternal Links:[Link](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Microsoft \(2024a\)MicrosoftIntfloat/multilingual\-e5\-base\.Note:HuggingFace model card\. Embedding model: 12 layers, embedding size 768External Links:[Link](https://huggingface.co/intfloat/multilingual-e5-base)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Microsoft \(2024b\)MicrosoftIntfloat/multilingual\-e5\-large\.Note:HuggingFace model card\. Embedding model: 24 layers, embedding size 1024; supports about 100 languagesExternal Links:[Link](https://huggingface.co/intfloat/multilingual-e5-large)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Microsoft \(2024c\)MicrosoftIntfloat/multilingual\-e5\-small\.Note:HuggingFace model cardExternal Links:[Link](https://huggingface.co/intfloat/multilingual-e5-small)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Mistral AI \(2024a\)Mistral AIMistralai/mistral\-7b\-instruct\-v0\.3\.Note:HuggingFace model cardExternal Links:[Link](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Mistral AI \(2024b\)Mistral AIMistralai/mistral\-nemo\-instruct\-2407\.Note:HuggingFace model card, developed with NVIDIAExternal Links:[Link](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Mistral AI \(2025\)Mistral AIMistralai/mistral\-small\-24b\-instruct\-2501\.Note:HuggingFace model cardExternal Links:[Link](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)Cited by:[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- OpenAI \(2024\)OpenAIHello gpt\-4o\.Note:OpenAIExternal Links:[Link](https://openai.com/index/hello-gpt-4o/)Cited by:[§3\.3](https://arxiv.org/html/2609.01810#S3.SS3.p3.1)\.
- OpenAI \(2025\)OpenAIIntroducing gpt\-4\.1 in the api\.Note:OpenAIExternal Links:[Link](https://openai.com/index/gpt-4-1/)Cited by:[§4\.1\.1](https://arxiv.org/html/2609.01810#S4.SS1.SSS1.p1.1)\.
- Papineniet al\.\(2002\)K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th annual meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p3.1)\.
- PDF2Go \(2025\)PDF2GoPDF2Go: convert pdf to text online\.Note:Online PDF conversion toolExternal Links:[Link](https://www.pdf2go.com/pdf-to-text)Cited by:[§3\.3](https://arxiv.org/html/2609.01810#S3.SS3.p2.1)\.
- Purwariantiet al\.\(2025\)A\. Purwarianti, D\. Adhista, A\. Baptiso, M\. Mahfuzh, Y\. Sabila, A\. Adila, S\. Cahyawijaya, and A\. F\. AjiNusaDialogue: dialogue summarization and generation for underrepresented and extremely low\-resource languages\.InProceedings of the Second Workshop in South East Asian Language Processing,pp\. 82–100\.Cited by:[§1](https://arxiv.org/html/2609.01810#S1.p1.1),[§2](https://arxiv.org/html/2609.01810#S2.p4.1)\.
- Rashkinet al\.\(2018\)H\. Rashkin, E\. M\. Smith, M\. Li, and Y\. BoureauTowards empathetic open\-domain conversation models: a new benchmark and dataset\.arXiv preprint arXiv:1811\.00207\.Cited by:[§1](https://arxiv.org/html/2609.01810#S1.p1.1),[§2](https://arxiv.org/html/2609.01810#S2.p1.1)\.
- Sadallahet al\.\(2025\)A\. Sadallah, J\. C\. Tonga, K\. Almubarak, S\. Almheiri, F\. Atif, C\. Qwaider, K\. Kadaoui, S\. Shatnawi, Y\. Alesh, and F\. KotoCommonsense reasoning in arab culture\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7695–7710\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p4.1)\.
- Shahid Beheshti University NLP Group \(2024\)Shahid Beheshti University NLP GroupSbunlp/fabert\.Note:HuggingFace model card\. Persian BERT\-base model \(FaBERT\) trained on HmBlogs corpusExternal Links:[Link](https://huggingface.co/sbunlp/fabert)Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p4.1),[§4](https://arxiv.org/html/2609.01810#S4.p2.1)\.
- Shamsfardet al\.\(2025\)M\. Shamsfard, Z\. Saaberi, S\. M\. H\. Hashemi, Z\. Vatankhah, M\. Ramezani, N\. Pourazin, T\. Zare, M\. Azimi, S\. Chitsaz, S\. Khoraminejad,et al\.FarsEval\-pkbets: a new diverse benchmark for evaluating persian large language models\.arXiv preprint arXiv:2504\.14690\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p4.1)\.
- Sureshet al\.\(2025\)S\. K\. Suresh, W\. Mengjun, T\. Pranav, and E\. S\. ChngDiasynth: synthetic dialogue generation framework for low resource dialogue applications\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 673–690\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p4.1)\.
- TakBook \(2025\)TakBookTakBook: free online book download platform\.Note:WebsiteExternal Links:[Link](https://www.takbook.com/)Cited by:[§3\.3](https://arxiv.org/html/2609.01810#S3.SS3.p2.1)\.
- Wikipedia contributors \(2009\)Wikipedia contributorsEssential articles every wikipedia should have\.Note:[https://fa\.wikipedia\.org](https://fa.wikipedia.org/)Cited by:[§3\.1](https://arxiv.org/html/2609.01810#S3.SS1.SSS0.Px1.p1.1)\.
- Wikipedia contributors \(2014\)Wikipedia contributorsMost viewed articles in persian wikipedia\.External Links:[Link](https://fa.wikipedia.org/)Cited by:[§3\.1](https://arxiv.org/html/2609.01810#S3.SS1.SSS0.Px1.p1.1)\.
- Wikipedia contributors \(2025a\)Wikipedia contributors100 essential articles \(persian wikipedia\)\.External Links:[Link](https://fa.wikipedia.org/wiki/Wikipedia:100_essential_articles)Cited by:[§3\.1](https://arxiv.org/html/2609.01810#S3.SS1.SSS0.Px1.p1.1)\.
- Wikipedia contributors \(2025b\)Wikipedia contributorsFeatured articles in persian wikipedia\.Note:[https://fa\.wikipedia\.org](https://fa.wikipedia.org/)Cited by:[§3\.1](https://arxiv.org/html/2609.01810#S3.SS1.SSS0.Px1.p1.1)\.
- Wuet al\.\(2019\)W\. Wu, Z\. Guo, X\. Zhou, H\. Wu, X\. Zhang, R\. Lian, and H\. WangProactive human\-machine conversation with explicit conversation goal\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 3794–3804\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p1.1)\.
- Xueet al\.\(2021\)L\. Xue, N\. Constant, A\. Roberts, M\. Kale, R\. Al\-Rfou, A\. Siddhant, A\. Barua, and C\. RaffelMT5: a massively multilingual pre\-trained text\-to\-text transformer\.InProceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies,pp\. 483–498\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p4.1)\.
- Zhanget al\.\(2023\)C\. Zhang, L\. D’Haro, C\. Tang, K\. Shi, G\. Tang, and H\. LiXDial\-eval: a multilingual open\-domain dialogue evaluation benchmark\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 5579–5601\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p1.1)\.
- Zhanget al\.\(2018\)S\. Zhang, E\. Dinan, J\. Urbanek, A\. Szlam, D\. Kiela, and J\. WestonPersonalizing dialogue agents: i have a dog, do you have pets too?\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2204–2213\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p1.1)\.
- Zhanget al\.\(2019\)T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. ArtziBertscore: evaluating text generation with bert\.arXiv preprint arXiv:1904\.09675\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p3.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p3.1)\.
- Zhouet al\.\(2020\)H\. Zhou, C\. Zheng, K\. Huang, M\. Huang, and X\. ZhuKdConv: a chinese multi\-domain dialogue dataset towards multi\-turn knowledge\-driven conversation\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 7098–7108\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p1.1)\.
- Zhouet al\.\(2018\)K\. Zhou, S\. Prabhumoye, and A\. W\. BlackA dataset for document grounded conversations\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 708–713\.Cited by:[§2](https://arxiv.org/html/2609.01810#S2.p2.1)\.

## Appendix AFull Dialogues ofWiki\-FaDial,DailyDialog\-FA, andPlayDial\-FA

Here, we present the complete dialogues from the three sources illustrated in Figure[1](https://arxiv.org/html/2609.01810#S0.F1):Wiki\-FaDial\(natural dialogues generated from Farsi Wikipedia paragraphs\),DailyDialog\-FA\(daily English conversations translated into culturally natural Farsi\), andPlayDial\-FA\(expressive multi\-turn dialogues extracted from Farsi plays\)\.

## Appendix BPrompt Used forWiki\-FaDialGeneration

##### System prompt:

This GPT model takes Farsi texts and transforms each one into a casual, informal dialogue between two people\. The generated dialogue should:

- •be natural, relatable, and culturally appropriate in Farsi,
- •include elements such as humor, honest reflection, light\-hearted frustration, empathetic advice, and small talk,
- •reflect personal or work\-related topics and daily routines when appropriate,
- •use natural idioms or slang and maintain an informal tone,
- •contain exactly 6 turns, each between 1 and 20 words,
- •include a"reference"field in each turn quoting or summarizing the relevant part of the source text for transparency\.

##### Input format \(JSON\):

```
{
  "texts": [
    "First Farsi text...",
    "Second Farsi text...",
    "Third Farsi text...",
    "Fourth Farsi text...",
    "Fifth Farsi text..."
  ]
}
```

##### Output format:

The model produces five independent JSON objects \(one per input text\)\. Each object contains only the generated dialogue and does not include the original input text:

```
{
  "dialogue": [
    {
      "speaker": "A",
      "text": "...",
      "reference": "..."
    },
    {
      "speaker": "B",
      "text": "...",
      "reference": "..."
    }
  ]
}
```

## Appendix CHuman Post\-editing Protocol forWiki\-FaDial

Below is the streamlined protocol followed by native\-speaker annotators when refiningWiki\-FaDialdialogues\. Two annotators work independently; a senior linguist resolves any conflicts\.

> Guideline:When editing or regenerating these dialogues, make sure each has exactly 6 short turns \(each 1–20 words\) in casual, friendly, natural Farsi, sounding like two people in a real\-life chat\. The language must be informal, sprinkled with humor, light frustration, empathy, or small talk, using idioms and spoken expressions that feel truly Iranian\. Each turn must include a reference that ties it clearly to the original text \(a short quote or idea summary\) to show transparency\. Ensure the conversation is not a dry summary but a playful, human back\-and\-forth — like two friends reacting, joking, giving quick advice, or sharing small relatable complaints\. After generating, carefully check the dialogue for all these points: if it fails \(too stiff, formal, robotic, or missing references\), regenerate it with a prompt asking for playful, informal style, or manually fix grammar, shorten long lines, replace unnatural words, or add emotional touches until it feels authentic\. Throughout, keep the JSON clean \(no trailing commas, correct brackets\), so the final output is short, vivid, idiomatic, well\-referenced, and technically valid\.

### Stage 1 – Coherence & Flow

- •Ensure each question follows logically from the previous answer\.
- •Delete turns that introduce off\-topic content or repeat earlier facts\.

### Stage 2 – Factuality & Reference

- •Check every statement against the source paragraph; replace or remove hallucinations\.
- •Verify names, dates, and numbers; correct diacritics\.

### Stage 3 – Style & Register

- •Keep each turn 1–20 tokens long; split or condense otherwise\.
- •Use informal Farsi: swap formal forms for idioms/slang; add discourse markers sparingly\.
- •Optional: insert light humour, empathy, or advice—without altering facts\.

### Stage 4 – Compliance & Safety

- •check Hate checks; anonymise or delete flagged text\.
- •Remove any content violating Wikimedia policy \(explicit, medical, self\-harm\)\.

## Reproducibility Details

##### Hardware\.

Experiments are conducted on3×\\timesNVIDIA RTX A6000GPUs \(48 GB each\) with mixed\-precision \(bf16\), FlashAttention, and gradient checkpointing\.

##### Software environment\.

Python 3\.11, PyTorch 2\.3\.0, HuggingFace Transformers≥\\geq0\.22, and BitsAndBytes 0\.45 for 4\-bit QLoRA\. All package versions are pinned inenvironment\.yml\.

##### Random seeds\.

We fixseed = 100for model initialisation, dataloader shuffling, and NumPy/PyTorch RNGs\. Results are averaged over three seeds for classification\.

##### Hyper\-parameters\.

- •*LLM fine\-tuning \(LoRA\):*per\-device batch size = 4, gradient accumulation = 2 \(effective batch size = 24 across 3 GPUs\), lr =×10−41\\\!\\times\\\!10^\{\-4\}\(AdamW, cosine decay\), weight decay =×10−41\\\!\\times\\\!10^\{\-4\}, warmup ratio = 0, max grad\-norm = 1\.0, epochs = 3, context = 512 tokens\. LoRA configuration: rank = 128,α\\alpha= 256, dropout = 0\.1\.
- •*Encoder classification \(Frozen encoder \+ MLP head\):*batch = 64, lr =×10−51\\\!\\times\\\!10^\{\-5\}\(AdamW\), epochs = 100, context = 256 tokens\. 4\-layer head: 512, 256, 128, 64→\\rightarrowLogSoftmax, NLLLoss\.
- •*Encoder classification \(Full fine\-tuning\):*batch = 32, lr =×10−52\\\!\\times\\\!10^\{\-5\}\(AdamW\), epochs = 3, context = 256 tokens\.

##### Code & data\.

All corpora, scripts, and checkpoints will be released\.

##### Compute footprint\.

LLM fine\-tuning consumes 16 GPU\-hours in total; encoder runs finish in∼\\sim2 hours per task\. This modest budget enables replication on most academic clusters\.

## Appendix DIllustrative Examples of Human Post\-editing forWiki\-FaDial

Below are five examples of human post\-editing forWiki\-FaDial\. Each example includes the original Wikipedia passage, the dialogue generated by the model, and the final version after manual revision by human annotators\.The following examples illustrate different types of interventions:

- •Example 1:The opening sentence contained a grammatical error that obscured the intended meaning, while the remainder of the dialogue was fluent and well\-structured\. As a result, only the first sentence was manually revised\.
- •Example 2:The model overused repetitive openers such as “Have you heard…” or “Have you read…”\. To increase conversational variety and avoid redundancy, annotators modified these introductions manually\.
- •Example 3:The generated sentences lacked syntactic coherence and logical progression, resulting in a disjointed and confusing dialogue\. Consequently, annotators chose to regenerate the entire dialogue using the model\.
- •Example 4:The model produced an inaccurate and unrealistic analogy—e\.g\., comparing a storm to a funnel—that was not grounded in the original source content\. This type of error, commonly seen in metaphorical or comparative contexts, was corrected by regenerating the dialogue, which yielded a coherent and accurate result on the second attempt\.
- •Example 5:No revisions were required\. The model\-generated dialogue was grammatically sound, culturally appropriate, and naturally reflected Farsi conversational style, even capturing a light sense of humor typical of everyday speech\.

## Appendix EModel Checkpoints

##### Large\-language models\.

1. 1\.meta\-llama/Llama\-3\.2\-1B\-Instruct
2. 2\.meta\-llama/Llama\-3\.2\-3B\-Instruct
3. 3\.meta\-llama/Llama\-3\.1\-8B\-Instruct
4. 4\.mistralai/Mistral\-7B\-Instruct\-v0\.3
5. 5\.mistralai/Mistral\-Nemo\-Instruct\-2407
6. 6\.mistralai/Mistral\-Small\-24B\-Instruct\-2501

##### Transformer encoders\.

1. 1\.intfloat/multilingual\-e5\-large
2. 2\.intfloat/multilingual\-e5\-base
3. 3\.intfloat/multilingual\-e5\-small
4. 4\.HooshvareLab/bert\-base\-parsbert\-uncased
5. 5\.sbunlp/fabert
6. 6\.google\-bert/bert\-base\-multilingual\-cased
7. 7\.FacebookAI/roberta\-base
8. 8\.FacebookAI/roberta\-large
9. 9\.FacebookAI/xlm\-roberta\-large
10. 10\.FacebookAI/xlm\-roberta\-base

## Translation Prompt

The following prompt was used to generate culturally authentic Farsi dialogues from English sources:

> Your task is to translate a given dialogue from English to Farsi, ensuring the following guidelines are followed: 1\. Farsi Names: Use Farsi names in the translated dialogue while maintaining the gender distinctions present in the English version\. 2\. Natural Language: Craft the conversation in everyday spoken Farsi, utilizing common expressions and adhering to cultural norms of politeness\. 3\. Engaging Flow: Ensure the dialogue consists of at least six exchanges between two characters\. Maintain a natural, logical, and engaging flow\. Where relevant, include elements such as questions, disagreements, or problem\-solving\. 4\. Cultural Authenticity: Reflect authentic Farsi cultural nuances in the conversation\. 5\. Dialogue Length: Keep the length of the dialogue the same as in the English version\. Do not add any new conversations or exchanges\. 6\. Output Format: Provide the complete conversation in a valid JSON format\. The dialogue should be entirely in Farsi, with no additional English commentary or context\. Maintain logical coherence and consistency throughout\. Expected JSON Output Format: ``` { "conversation": [ { "speaker": "first person", "text": "{{response_1}}" }, ... ] } ``` English dialogue:dialog

## Appendix FHuman Post\-editing Protocol forDailyDialog\-FA

Below is the streamlined protocol followed by native\-speaker annotators when refiningDailyDialog\-FAdialogues\. Two annotators work independently to post\-edit the model\-generated Farsi dialogues to ensure naturalness, cultural relevance, and structural fidelity to the original English source\.

### Editing Procedure

Once the model generates the Farsi dialogue from English input, human annotators follow this step\-by\-step process:

1. 1\.Fluency and Naturalness Ensure that each utterance reads like natural, everyday spoken Farsi\. Sentences should flow conversationally, avoiding overly literal translations or stiff, unnatural phrasing\. Aim for speech that reflects how native speakers actually talk\.
2. 2\.Cultural Localization Adapt all cultural elements to fit an Iranian context\. This includes: - •Replacing foreign currencies \(e\.g\., USD, Euro\) with the appropriate Iranian currency\. - •Substituting references to globally known locations \(e\.g\., Niagara Falls\) with locally recognizable or culturally familiar places\. - •Adjusting units of measurement or public references \(e\.g\., holidays, social norms\) to culturally meaningful equivalents\.
3. 3\.Preserving Dialogue Structure Keep the number of dialogue turns exactly the same as the original English version\. Do not split, merge, or remove turns unless absolutely necessary for clarity or fluency\. Preserve the logical progression and intention behind each exchange\.
4. 4\.Speaker Identity and Naming Assign culturally appropriate Farsi names that match the gender and style of the original English characters\. Avoid names that may seem foreign or uncommon in the Iranian context\. Gender agreement between speakers’ names and pronouns must be consistent throughout\.
5. 5\.Grammatical and Semantic Corrections Correct any errors in grammar, word choice, or sentence structure\. Fix mistranslations that may alter or obscure the intended meaning\. Pay special attention to: - •Verb tense consistency - •Proper use of prepositions and conjunctions - •Subject\-verb agreement - •Pronoun resolution
6. 6\.Removal of Artifacts Eliminate any residual English words, incomplete phrases, formatting markers, or annotation artifacts \(e\.g\., brackets, placeholder tokens\)\. Ensure that the dialogue reads cleanly, without machine\-generated traces\.
7. 7\.Tone and Politeness Maintain an appropriate register for informal dialogue\. The tone should feel friendly, polite, and contextually appropriate for casual interaction\. Avoid archaic, poetic, or overly formal language unless explicitly present in the original\.
8. 8\.Formatting and Output Ensure the final dialogue is submitted in valid JSON format, with clearly structured key\-value pairs for each turn\. No syntax errors should be present, and spacing or punctuation should follow standard conventions\.

## Appendix GIllustrative Examples of Human Post\-editing forDailyDialog\-FA

Below are five representative examples fromDailyDialog\-FA\. Each example includes the original English dialogue, the raw Farsi translation generated by the model, and the final version after human post\-editing\. These examples illustrate the types of edits performed and the reasoning behind them\.

- •Example 1:References to “dollars” were converted to “tomans” to reflect the local currency used in Iran\.
- •Example 2:A grading scale out of 100 was adapted to a 0–20 scale, which is standard in the Iranian education system\.
- •Example 3:Mention of China’s Huangguoshu Waterfall was replaced with Iran’s Laton Waterfall to create a culturally familiar context\.
- •Example 4:An American store name was substituted with a well\-known Iranian brand\.
- •Example 5:The model autonomously replaced an American personal name with a natural\-sounding Farsi name, requiring no human edits\.

Each case illustrates the specific types of transformations necessary to produce high\-quality, culturally grounded dialogues that are suitable for native Farsi speakers\.

## Appendix HSegmentation prompt

### Task Description

Your duty is to receive the text and segment it strictly based on the rules below\. The output must be inFarsiand presented in aclear, organized table format\.

### Segmentation Rules

1. 1\.Scene Continuity: - •Strictly follow scene continuity\. - •Start a new segment only when there is anexplicit scene change, such as: - –Change inlocation - –Change intime - –A significant shift in theconversation context\(not a minor one\)
2. 2\.Dialogue Preservation: - •Preserve the original dialoguesexactly as they appear in the text\. - •Donotomit, alter, summarize, or paraphrase any part\. - •Keep interruptions, incomplete sentences, repetitions, and pauses\. - •Clearly indicate thespeakerfor each line\. - •Double\-check for completeness to ensure no dialogue lines are missing after segmentation\.
3. 3\.Output Format: - •Present the output in a table with the following columns: 1. \(a\)Title:A short, descriptive title summarizing the essence of the scene\. 2. \(b\)Characters:List all characters who have dialogues or are mentioned in the segment\. 3. \(c\)Dialogues:Present all dialogues exactly as in the original script, clearly marking each speaker\. 4. \(d\)Overall Sentiment:Choose fromPositive,Negative, orNeutral\. 5. \(e\)Reference:Always write the name of the theatre play\.

### Additional Instructions

- •Ensure each scene iscompletebefore moving to the next one\.
- •Donotomit any part of the text\.
- •Double\-check for any missing dialogues after segmentation\.
- •Maintain clarity and organization for easy analysis\.

## Appendix IAnnotation and Editing Protocol for Play\-Based Dialogues

To ensure consistency and quality during corpus construction, annotators followed a structured protocol while cleaning, extracting, reformulating, and validating dialogue segments from the Farsi play scripts\.

- •Remove non\-dialogue content\.Delete stage directions, narrator descriptions, scene headers, page numbers, and formatting artifacts introduced by OCR\. Only spoken dialogue lines should remain\.
- •Preserve original dialogue content\.The objective of the extraction stage is to retain the spoken dialogue exactly as it appears in the original plays\. Annotators must not paraphrase, rewrite, or introduce new content at this stage\.
- •Correct OCR artifacts\.Fix obvious OCR errors such as broken characters, incorrect punctuation, or merged tokens while preserving the intended wording of the original script\.
- •Maintain speaker attribution\.Ensure that each dialogue line remains associated with the correct character in the original script\. If OCR errors obscure the speaker name, the annotator should verify it by consulting the original PDF\.
- •Preserve conversational order\.Dialogue lines must remain in the exact sequence in which they appear in the original play\. Reordering dialogue turns is not permitted\.
- •Remove stage interruptions within dialogue\.If stage directions appear inside dialogue lines \(e\.g\., gestures, pauses, or descriptions of actions\), remove them while keeping the surrounding spoken text intact\.
- •Verify extraction against the original script\.After cleaning the OCR output, annotators compare the extracted dialogue with the original play to ensure that no dialogue lines were accidentally omitted or altered\.
- •Check scene boundaries for segmentation\.When scripts are segmented into dialogue scenes, ensure that a new dialogue begins only when the scene clearly changes in the original play\. Segmentation should preserve the natural conversational flow within each scene\.
- •Verify reformulated dialogues for copyright compliance\.After segmentation, dialogue segments are reformulated usingGPT\-4oto avoid verbatim reproduction of copyrighted play scripts\. Annotators compare each reformulated dialogue with the original scene to ensure that the conversational intent, speaker interactions, and thematic content are preserved while avoiding direct copying of the source text\.
- •Preserve conversational structure\.During reformulation verification, annotators ensure that the number of turns and the overall conversational flow remain consistent with the original scene\.
- •Prevent verbatim overlap\.Annotators ensure that reformulated dialogues do not contain long verbatim spans from the original plays, thereby preventing redistribution of copyrighted text\.
- •Flag ambiguous or corrupted cases\.If OCR errors, missing dialogue, or unclear speaker attribution are encountered, annotators flag the issue for discussion and resolve it during the adjudication stage\.
- •Validate sentiment labels\.For sentiment annotation, annotators review the dialogue within the context of the original scene to determine whether the overall sentiment is*positive*,*neutral*, or*negative*\. When disagreements occur, annotators discuss the case and finalize the label through consensus\.

## Appendix JExample of Segmentation, Reformulation, and Human Verification

This appendix provides an illustrative example of the pipeline used to construct dialogues inPlayDial\-FA\. For each scene, the original script excerpt is first segmented into a dialogue, then reformulated byGPT\-4oto avoid verbatim copying, and finally verified and corrected by human annotators\. See pages \- of[namyeshname\.pdf](https://namyeshname.pdf/)

## Appendix KHuman Evaluation Guidelines

To assess the quality of generated dialogues inWiki\-FaDial, we conducted a human evaluation study with native Farsi speakers\. Annotators evaluated each generated dialogue independently using a five\-level quality scale designed to measure conversational naturalness, coherence, fluency, and contextual appropriateness\.

Annotators were instructed to read the full dialogue and assign a single overall score according to the following criteria:

- •A— Highly natural, coherent, fluent, and contextually appropriate dialogue\. Responses resemble realistic human conversation with clear conversational flow and culturally natural phrasing\.
- •B— Good\-quality dialogue with only minor grammatical, fluency, or phrasing issues\. The dialogue remains natural, understandable, and contextually consistent\.
- •C— Acceptable dialogue with noticeable problems in fluency, coherence, or naturalness, but still understandable overall\.
- •D— Weak dialogue containing major inconsistencies, unnatural responses, repetitive phrasing, or poor conversational flow\.
- •E— Very poor dialogue that is incoherent, irrelevant, broken, or fails to maintain meaningful conversation\.

Annotators were instructed to consider the following dimensions during evaluation:

1. 1\.Fluency:grammatical correctness and readability of individual utterances\.
2. 2\.Coherence:logical consistency and continuity across dialogue turns\.
3. 3\.Contextual Appropriateness:relevance of responses to the conversational context\.
4. 4\.Naturalness:similarity to realistic everyday Farsi conversation\.
5. 5\.Cultural Plausibility:use of culturally appropriate expressions, politeness conventions, and conversational style\.

Annotators were encouraged to evaluate dialogues holistically rather than penalizing isolated minor mistakes\. Dialogues with occasional grammatical errors could still receive high scores if the overall interaction remained coherent and natural, while dialogues with strong lexical overlap but unnatural conversational structure were assigned lower scores\.

Below, we provide representative examples corresponding to all five evaluation categories \(A–E\)\. See pages \- of[example\.pdf](https://example.pdf/)

## Appendix LIndependent External Validation

To further assess dataset quality and annotation reliability, we conducted an independent validation study using a native Farsi speaker who was not involved in dataset construction or verification\. The annotator evaluated stratified random samples of 100 instances for each annotation task, resulting in 300 evaluation items across the TalkFa datasets\.

### L\.1Subjective Quality Evaluation

The annotator evaluated translation and generation quality using Likert\-style scales ranging from 1–3 or 1–4 depending on the evaluation criterion\. Table[10](https://arxiv.org/html/2609.01810#A12.T10)summarizes the results\.

DatasetMetricMeanScaleInterpretationWiki\-FaDialFaithfulness3\.003Fully faithful to source textNaturalness3\.604Highly natural phrasingCultural Appropriateness2\.603Mostly culturally appropriateLLM Artifacts2\.603Very few obvious artifactsDailyDialog\-FAMeaning Preservation3\.003Fully faithful translationNaturalness4\.004Fully natural spoken FarsiCultural Localization2\.953Strong localization qualityPlayDial\-FANaturalness3\.304Natural theatrical dialogueTable 10:Independent external evaluation of subjective dialogue quality across the TalkFa datasets\.
### L\.2Inter\-Annotator Agreement

We further computed raw agreement and Cohen’sκ\\kappabetween the independent annotator and the original gold annotations\.

Table 11:Agreement between the independent annotator and the original gold annotations\. Agreement interpretation follows Landis and Koch[Landis and Koch \(1977\)](https://arxiv.org/html/2609.01810#bib.bib53)\.
### L\.3Discussion

The independent external validation provides additional evidence for the quality and reproducibility of the TalkFa benchmark\. Across 300 stratified random samples \(100 per annotation task\), the independent native Farsi annotator assigned high quality scores to all three datasets\.Wiki\-FaDialachieved 2\.8/3\.00 for factual grounding, 3\.65/4\.00 for naturalness, and 2\.70/3\.00 for cultural appropriateness\.DailyDialog\-FAobtained perfect meaning preservation \(3\.00/3\.00\), high naturalness \(3\.90/4\.00\), and 2\.8/3\.00 for cultural localization, whilePlayDial\-FAachieved 3\.50/4\.00 for dialogue naturalness\. These results indicate that the combination of LLM\-assisted construction and multi\-stage native\-speaker verification produces fluent, culturally appropriate dialogues while preserving the intended semantic content\. Annotation reproducibility was likewise consistently high\. Dialogue\-act annotation achieved 85% raw agreement with Cohen’sκ=0\.800\\kappa=0\.800, emotion annotation reached the highest agreement with 89% \(κ=0\.872\\kappa=0\.872\), and sentiment annotation achieved 87% \(κ=0\.805\\kappa=0\.805\)\. Despite the greater subjectivity of emotion and sentiment interpretation, the substantial to almost perfect agreement across all tasks demonstrates that the released annotations are reliable and can be consistently reproduced by an independent annotator\. Overall, the external validation complements the construction\-time quality control by confirming both the linguistic quality of the dialogues and the robustness of the released annotations, providing additional evidence for the reliability of the TalkFa benchmark\.

## Appendix MData\-Efficiency Ablation

To assess data efficiency, we repeat LoRA fine\-tuning using\{25%,50%,75%,100%\}\\\{25\\%,50\\%,75\\%,100\\%\\\}of theWiki\-FaDialtraining set and evaluate performance using BERTCos and ROUGE\-L\.

Figure[4](https://arxiv.org/html/2609.01810#A13.F4)shows that performance improves sharply from the Base model to 25% of the training data, followed by diminishing returns\. Across all model sizes, the first quarter of the training set recovers more than 90% of final performance gains, indicating that relatively small grounded dialogue corpora can effectively teach the task\. However, despite these gains, substantial gaps to human\-quality dialogue remain, as discussed in Sec\.[4\.1\.1](https://arxiv.org/html/2609.01810#S4.SS1.SSS1)\.

Figure 4:ParsBERT cosine \(top\) and ROUGE\-L F1 \(bottom\) versus fraction of the training data\.
## Appendix NAutomatic vs\. Human Evaluation

Table[12](https://arxiv.org/html/2609.01810#A14.T12)summarizes the disconnect between automatic metrics and human\-centered evaluation\.

Table 12:Comparison between automatic metrics and human\-centered evaluation\. ✗ indicates that the metric overestimates perceived dialogue quality; ✓ indicates a more informative quality signal\.The disparity between high automatic scores \(≥0\.93\\geq 0\.93\) and moderate human ratings \(2\.74/5\) indicates that semantic overlap metrics alone fail to capture dialogue naturalness, coherence, and contextual appropriateness, making human or LLM\-based evaluation necessary for calibrated assessment\.