When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates

arXiv cs.CL Papers
arabic hebrew cognate-benchmark multilingual llm semantic-understanding cross-lingual

Summary

This paper introduces SemCog Bench, a curated benchmark of 1,858 Arabic-Hebrew word pairs with sentence-level annotations, to evaluate LLMs' ability to distinguish true cognates from false friends and loanwords. Results show high accuracy on true cognates but sharp drops on false friends, highlighting a key limitation in cross-lingual semantic reasoning.

arXiv:2606.13218v1 Announce Type: new Abstract: Arabic and Hebrew, as closely related Semitic languages, share a substantial lexicon of true cognates, misleading false friends, and modern loanwords. This overlap poses a challenge for cross-lingual semantic understanding in large language models (LLMs). To evaluate this capability, we introduce SemCog Bench, a curated benchmark of 1,858 Arabic--Hebrew word pairs with sentence-level annotations for cognate identification and semantic disambiguation. We evaluate open-source and commercial LLMs across multiple input representations (raw, diacritized, Romanized, and phonetic) and reveal a critical gap in cross-lingual reasoning. While models achieve high accuracy on true cognates, performance drops sharply on false friends and loanwords, reflecting a strong reliance on surface-form similarity. Furthermore, sentence-level context yields only modest improvements, suggesting that contextual cues alone are insufficient to overcome misleading form-based signals. These findings reveal a fundamental limitation of current LLMs in resolving cross-lingual form--meaning conflicts and establish SemCog Bench as a rigorous benchmark for multilingual semantic reasoning. Our code and data are publicly available.
Original Article
View Cached Full Text

Cached at: 06/12/26, 08:52 AM 

# When Similar Means Different: Evaluating LLMs on Arabic--Hebrew Cognates
Source: [https://arxiv.org/abs/2606.13218](https://arxiv.org/abs/2606.13218)
[View PDF](https://arxiv.org/pdf/2606.13218)

> Abstract:Arabic and Hebrew, as closely related Semitic languages, share a substantial lexicon of true cognates, misleading false friends, and modern loanwords\. This overlap poses a challenge for cross\-lingual semantic understanding in large language models \(LLMs\)\. To evaluate this capability, we introduce SemCog Bench, a curated benchmark of 1,858 Arabic\-\-Hebrew word pairs with sentence\-level annotations for cognate identification and semantic disambiguation\. We evaluate open\-source and commercial LLMs across multiple input representations \(raw, diacritized, Romanized, and phonetic\) and reveal a critical gap in cross\-lingual reasoning\. While models achieve high accuracy on true cognates, performance drops sharply on false friends and loanwords, reflecting a strong reliance on surface\-form similarity\. Furthermore, sentence\-level context yields only modest improvements, suggesting that contextual cues alone are insufficient to overcome misleading form\-based signals\. These findings reveal a fundamental limitation of current LLMs in resolving cross\-lingual form\-\-meaning conflicts and establish SemCog Bench as a rigorous benchmark for multilingual semantic reasoning\. Our code and data are publicly available\.

## Submission history

From: Bashar Alhafni \[[view email](https://arxiv.org/show-email/263cd545/2606.13218)\] **\[v1\]**Thu, 11 Jun 2026 11:33:09 UTC \(3,835 KB\)

Similar Articles

Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth

arXiv cs.CL

This paper introduces a cross-evaluation framework for benchmarking LLMs on Arabic cultural and sociolinguistic knowledge, using human SME ground truth and automated judges. The authors contribute a dataset of prompt-rubric pairs for Egyptian and Iraqi Arabic, evaluating frontier LLMs and finding that cultural reasoning remains a primary failure mode for automated grading.

Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs

arXiv cs.CL

This paper proposes a semantic verification framework using Natural Language Inference (NLI) to evaluate the sensitivity of clinical LLMs to meaning-preserving prompt variations, introducing metrics such as MVS, ΔC, and WCI. Results show that domain specialization does not consistently improve robustness, with both domain-specific and general-purpose models showing mixed performance.