E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets

arXiv cs.CL Papers
arabic-nlp textual-entailment benchmarks datasets evaluation

Summary

The paper introduces E-CONAN benchmarks for Arabic textual entailment and natural inference, featuring diverse datasets and evaluations of multilingual and LLM models to enrich research in the domain.

arXiv:2609.11334v1 Announce Type: new Abstract: Natural Language Inference processes pairs of sentences to extract their semantic relations. NLI has been a hot research topic, integrated as a main component in other NLP applications. Despite significant advancements in textual inference across various languages all around the world, Arabic language still suffers from limited resources in this domain. To address this gap, this paper introduces E-CONAN benchmarks that are composed of sentences pairs from various sources: (1) automatically-translated pairs, (2) human-validated machine-translated pairs, (3) hand-crafted pairs from teaching Arabic as foreign language books, and (4) headlines pairs from different news channels containing rumors. E-CONAN contains two benchmark datasets, E-CONAN-2, a 2-way dataset (RTE) and E-CONAN-3, a 3-way dataset (NLI). Additionally, we have used E-CONAN benchmarks to evaluate 9 state-of-the-art multilingual pretrained models using zero-shot classification. Models were evaluated across the ArNLI, XNLI, and E-CONAN datasets. Results show that E-CONAN is a potentially valuable resource for evaluating model generalization and even for fine-tuning pre-trained models. Its diverse composition, derived from a combination of sources, offers a broader and more robust assessment compared to XNLI and ArNLI. In addition, we have evaluated 5 LLMs on E-CONAN-3 dataset. Moreover, we incorporated MARBERT as a representative Arabic-specific baseline and conducted performance evaluation comparison to demonstrate how Arabic-specific models scale against cross-lingual and LLM-based approaches on the E-CONAN benchmarks. Furthermore, we conducted detailed qualitative and quantitative error analysis to analyze frequent error patterns. E-CONAN benchmarks will be publicly available, we hope that it will enrich research community in Arabic textual entailment and natural language inference.
Original Article
View Cached Full Text

Cached at: 09/11/26, 08:28 AM 

# E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets
Source: [https://arxiv.org/abs/2609.11334](https://arxiv.org/abs/2609.11334)
[View PDF](https://arxiv.org/pdf/2609.11334)

> Abstract:Natural Language Inference processes pairs of sentences to extract their semantic relations\. NLI has been a hot research topic, integrated as a main component in other NLP applications\. Despite significant advancements in textual inference across various languages all around the world, Arabic language still suffers from limited resources in this domain\. To address this gap, this paper introduces E\-CONAN benchmarks that are composed of sentences pairs from various sources: \(1\) automatically\-translated pairs, \(2\) human\-validated machine\-translated pairs, \(3\) hand\-crafted pairs from teaching Arabic as foreign language books, and \(4\) headlines pairs from different news channels containing rumors\. E\-CONAN contains two benchmark datasets, E\-CONAN\-2, a 2\-way dataset \(RTE\) and E\-CONAN\-3, a 3\-way dataset \(NLI\)\. Additionally, we have used E\-CONAN benchmarks to evaluate 9 state\-of\-the\-art multilingual pretrained models using zero\-shot classification\. Models were evaluated across the ArNLI, XNLI, and E\-CONAN datasets\. Results show that E\-CONAN is a potentially valuable resource for evaluating model generalization and even for fine\-tuning pre\-trained models\. Its diverse composition, derived from a combination of sources, offers a broader and more robust assessment compared to XNLI and ArNLI\. In addition, we have evaluated 5 LLMs on E\-CONAN\-3 dataset\. Moreover, we incorporated MARBERT as a representative Arabic\-specific baseline and conducted performance evaluation comparison to demonstrate how Arabic\-specific models scale against cross\-lingual and LLM\-based approaches on the E\-CONAN benchmarks\. Furthermore, we conducted detailed qualitative and quantitative error analysis to analyze frequent error patterns\. E\-CONAN benchmarks will be publicly available, we hope that it will enrich research community in Arabic textual entailment and natural language inference\.

## Submission history

From: Khloud Al Jallad \[[view email](https://arxiv.org/show-email/1ec5181f/2609.11334)\] **\[v1\]**Thu, 10 Sep 2026 10:11:36 UTC \(1,684 KB\)

Similar Articles

Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth

arXiv cs.CL

This paper introduces a cross-evaluation framework for benchmarking LLMs on Arabic cultural and sociolinguistic knowledge, using human SME ground truth and automated judges. The authors contribute a dataset of prompt-rubric pairs for Egyptian and Iraqi Arabic, evaluating frontier LLMs and finding that cultural reasoning remains a primary failure mode for automated grading.