ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs
Summary
Introduces ALEE, a framework that uses Abstract Meaning Representations to generate English minimal pairs with controlled semantic shifts and translates them for evaluating text embeddings across 275+ languages, revealing persistent gaps in cross-lingual semantic representation.
View Cached Full Text
Cached at: 07/02/26, 05:36 AM
# ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs Source: [https://arxiv.org/abs/2607.00171](https://arxiv.org/abs/2607.00171) [View PDF](https://arxiv.org/pdf/2607.00171) > Abstract:Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge\. Current benchmarks are static, cover only a limited set of languages, are often domain\-specific, susceptible to overfitting, and poorly representative of low\-resource languages\. To address these limitations, we introduce ALEE, a framework that extends Sentence Smith \(Li et al\., 2025\) to the cross\-lingual and paragraph level\. ALEE uses Abstract Meaning Representations \(AMR\) to generate English minimal pairs with controlled, fine\-grained semantic shifts, which are paired with translations in target languages\. This approach enables targeted diagnostics for models in any language with English parallel data\. We conduct a large\-scale empirical study across a diverse set of embedding models and 275\+ languages spanning three parallel datasets\. On ALEE, performance varies substantially across languages, text lengths, and linguistic phenomena, exposing persistent gaps in cross\-lingual semantic representation that track language prevalence in training resources and subword tokenization\. We release ALEE at[this https URL](https://github.com/Andrian0s/any-lang-embed-eval) ## Submission history From: Andrianos Michail \[[view email](https://arxiv.org/show-email/c8a3d487/2607.00171)\] **\[v1\]**Tue, 30 Jun 2026 20:45:17 UTC \(2,903 KB\)
Similar Articles
Discovering Lexical Gaps Using Embeddings from Multilingual LLMs
This paper proposes a data-driven framework using embeddings from multilingual LLMs to detect lexical gaps between languages, achieving high accuracy in Korean-English pairs.
Multilingual Sentence Embeddings for Linguistic-Integrated Reliability Audit
This paper evaluates whether multilingual sentence embeddings can replace translation for linguistic-integrated reliability auditing across multiple languages in educational assessments, finding that native-language embeddings reproduce translation-based reliability estimates closely.
Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs
This paper investigates cross-lingual alignment in small Japanese-English bilingual language models by probing cultural and pragmatic reasoning, introducing the J-PragEval-v0 benchmark and proposing Pragmatic Representation Steering for inference-time interventions.
Evaluating Multilingual Sentence Embeddings for Translation Error Detection:An English--Greek Contrastive Study
This study evaluates multilingual sentence embeddings for distinguishing correct English–Greek translations from erroneous ones, finding that embeddings provide useful semantic signals but are better integrated into broader translation evaluation frameworks.
SemBridge: Language Transfer in Sparse Encoders via Multilingual Semantic Bridges
SemBridge is a novel embedding initialization method that leverages multilingual bridge models to establish semantic alignments between source and target vocabularies, improving cross-lingual sparse encoder adaptation and retrieval performance across multiple languages.