UOL@IDEM at BEA 2026 Shared Task 1: Neural Fusion and Feature-Rich Modeling for L1-Aware Vocabulary Difficulty Prediction

# UOL@IDEM at BEA 2026 Shared Task 1: Neural Fusion and Feature-Rich Modeling for L1-Aware Vocabulary Difficulty Prediction
Source: [https://arxiv.org/html/2606.24501](https://arxiv.org/html/2606.24501)
Nouran Khallaf1,2Serge Sharoff1 1Centre for Translation, Localisation and Interpreting Studies, School of Languages, Cultures and Societies, University of Leeds, UK 2Alexandria University, Egypt \{n\.khallaf,s\.sharoff\}@leeds\.ac\.uk

###### Abstract

This paper describes UOL@IDEM’s closed\-track submission to the BEA 2026 shared task on L1\-aware vocabulary difficulty prediction111[https://github\.com/Nouran\-Khallaf/UoL\-IDEM\-BEA2026\-Vocabulary\-Difficulty\-Prediction](https://github.com/Nouran-Khallaf/UoL-IDEM-BEA2026-Vocabulary-Difficulty-Prediction)\. We model the task as regression and train separate systems for Spanish, German, and Mandarin Chinese222Below we use*Chinese*for brevity\.\. Our system combines multilingual contextual representations with engineered features capturing frequency, surface form, retrieval evidence, semantic alignment, cognate similarity, and masked\-language\-model predictability\. Development results show consistent gains over the official closed\-track baselines, with sentence\-embedding encoders such as BGE\-M3, multilingual E5, and LaBSE performing best\. Official submissions achieve RMSE scores of 1\.132, 1\.037, and 0\.891 for Spanish, German, and Chinese, respectively\. Feature analysis identifies frequency as the most stable predictor, while contextual predictability, form similarity, retrieval, and semantic features provide complementary L1\-sensitive signals\. Error analysis shows strong ranking performance but weaker calibration for the easiest items, which are often overpredicted\.

UOL@IDEM at BEA 2026 Shared Task 1: Neural Fusion and Feature\-Rich Modeling for L1\-Aware Vocabulary Difficulty Prediction

Nouran Khallaf1,2Serge Sharoff11Centre for Translation, Localisation and Interpreting Studies,School of Languages, Cultures and Societies, University of Leeds, UK2Alexandria University, Egypt\{n\.khallaf,s\.sharoff\}@leeds\.ac\.uk

## 1Introduction

Text complexity and readability assessment are central to educational NLP, language learning, and text adaptation\. A key component is lexical complexity prediction \(LCP\), which estimates how difficult a word or multi\-word expression is likely to be for a target reader\(Shardlowet al\.,[2021](https://arxiv.org/html/2606.24501#bib.bib11),[2022](https://arxiv.org/html/2606.24501#bib.bib12); Northet al\.,[2023](https://arxiv.org/html/2606.24501#bib.bib13)\)\. Lexical difficulty is shaped by frequency, word length, morphology, polysemy, context, and reader background, and supports applications such as readability assessment, simplification, tutoring, and machine translation\(Shardlowet al\.,[2022](https://arxiv.org/html/2606.24501#bib.bib12); Northet al\.,[2023](https://arxiv.org/html/2606.24501#bib.bib13); Ohuobaet al\.,[2024](https://arxiv.org/html/2606.24501#bib.bib36)\)\.

The BEA 2026 Shared Task extends LCP by making vocabulary difficulty explicitly L1\-aware\. In this setting, the difficulty of an English target word depends not only on the word itself, but also on the context of its use as well as on its relationship to the learner’s first language through translation and transfer cues\. As a result, the same English word may have different difficulty profiles for Spanish, German, and Chinese speakers\. More specifically, the task uses an extended version of the Knowledge\-based Vocabulary Lists \(KVL\) resource, pairing English target items with multilingual prompts and psychometrically calibrated GLMM\-based difficulty scores\(Skidmoreet al\.,[2025](https://arxiv.org/html/2606.24501#bib.bib3)\)\.

In the closed track, systems train separate models for each L1 using only the shared\-task resources\(Felice and Skidmore,[2026](https://arxiv.org/html/2606.24501#bib.bib2); British Council,[2026](https://arxiv.org/html/2606.24501#bib.bib1)\)\. We therefore model the task as three parallel prediction settings: Spanish→\\rightarrowEnglish, German→\\rightarrowEnglish, and Chinese→\\rightarrowEnglish\.

Our approach experiments with mid\-sized text encoders \(of up to 560M parameters\) enriched with engineered features, such as lexical frequency, orthographic clues, contextual retrieval, cognate\-like similarity, semantic\-domain alignment, and masked\-language\-model predictability\. We evaluate several encoders within a neural fusion framework and compare full\-feature and reduced\-feature variants\.

Our experiments show three main findings\. First, neural fusion improves over the official closed\-track baselines across all three languages, improving RMSE by up to 0\.21–0\.26\. Second, sentence\-embedding\-oriented multilingual models, especially BGE\-M3\(Chenet al\.,[2024](https://arxiv.org/html/2606.24501#bib.bib9)\), multilingual E5\(Wanget al\.,[2024](https://arxiv.org/html/2606.24501#bib.bib7)\), and LaBSE\(Fenget al\.,[2022](https://arxiv.org/html/2606.24501#bib.bib6)\), are more effective than plain transformer baselines such as mBERT and XLM\-R\. Third, feature and error analyses show that frequency is the most stable predictor contributing above the encoder baseline, while semantic, cognate, retrieval, and MLM\-based cues can provide complementary L1\-sensitive information\.

## 2Data and Preprocessing

Data exploration showed thatL1\_source\_wordoften contains noisy forms that require language\-aware normalisation before feature extraction\. We address lexical variants, negative constraints, morphological fragments, negated instructional patterns, explanatory glosses, and punctuation\. For example, multiple alternatives are split and tracked with a variant\-count feature \(Korridor, Flur→\\rightarrowKorridor\); exclusion notes are removed and encoded as binary features \(Flugzeug \(nicht: …\)→\\rightarrowFlugzeug\); truncated German compounds are normalised to stems \(Regierungs\-→\\rightarrowRegierungs\); Spanish instruction\-like phrases such asno esandla respuesta no termina enare stripped to isolate the lexical item; and Chinese punctuation, leading glosses, and parenthetical notes are pruned where appropriate \(（美）走廊，过道→\\rightarrow走廊\)\. To make preprocessing traceable, we keep the original source\-word string and use a cleaned lexical form for feature extraction\. When an English target contains multiple candidates, we select the candidate that appears in the L1 context\.

## 3Methodology

We formulated the task as supervised regression\. Each instance contained an L1\-specific context, an L1 source word, an English target word, and a continuous difficulty score\. Models used textual input and engineered features capturing complementary cues: frequency, lexical form, retrieval evidence, MLM and surprisal estimates, semantic\-domain information, and cognate\-based similarity\.

We compared three integration strategies: \(1\) textualised feature regression, which renders selected features as part of the input prompt; \(2\) multi\-stage late fusion, which combines frozen encoder embeddings with downstream regressors; and \(3\) neural fusion, which jointly optimises contextual representations and structured tabular features\. Because neural fusion performed best in the initial comparison, we used it as the main setting\.

### 3\.1Engineered features

In addition to BERT\-derived text representations, we designed a set of engineered features to capture lexical difficulty, cross\-lingual retrievability, contextual predictability, semantic compatibility, and source\-word ambiguity\. We group these features into frequency\-based, lexical, retrieval\-based, masked\-language\-model and surprisal, semantic, and cognate\-based features\. Full feature descriptions are provided in Appendix[A](https://arxiv.org/html/2606.24501#A1), TableLABEL:tab:appendix\_engineered\_features\.

#### Frequency\-based features\.

We combine three complementary frequency resources\. KELLY\(Kilgarriffet al\.,[2014](https://arxiv.org/html/2606.24501#bib.bib21)\)is derived from curated Web\-corpora and provides pedagogically oriented indicators, including frequency rank, percentile, and CEFR\-aligned labels, which approximate learner vocabulary progression\(Council of Europe,[2001](https://arxiv.org/html/2606.24501#bib.bib22)\)\. ThewordfreqPython package provides general corpus\-based estimates, including raw frequency, Zipf frequency, frequency percentile, and inverse\-frequency cost\(Speeret al\.,[2018](https://arxiv.org/html/2606.24501#bib.bib23); Sharoffet al\.,[2017](https://arxiv.org/html/2606.24501#bib.bib26)\)\. The SUBTLEX list is derived from subtitles and provides frequency and contextual\-diversity measures, reflecting both occurrence frequency and exposure across everyday contexts\(Brysbaert and New,[2009](https://arxiv.org/html/2606.24501#bib.bib24); van Heuvenet al\.,[2014](https://arxiv.org/html/2606.24501#bib.bib25)\)\. Combined, these features model familiarity from pedagogical, general\-corpus, and everyday\-exposure perspectives, providing a strong baseline signal for vocabulary difficulty prediction\(Shardlowet al\.,[2021](https://arxiv.org/html/2606.24501#bib.bib11),[2022](https://arxiv.org/html/2606.24501#bib.bib12); Northet al\.,[2023](https://arxiv.org/html/2606.24501#bib.bib13)\)\.

#### Surface\-form and morphosyntactic features\.

We extract surface\-form and morphosyntactic features from the English target and cleaned L1 source word\. These include target/source length, target syllable count, clue informativeness, and hidden clue characters\. We also encode POS information, using the dataset\-provided target POS and spaCy\-derived source POS\(Honnibalet al\.,[2020](https://arxiv.org/html/2606.24501#bib.bib27)\)\. Alongside these features, we also retain a set of preprocessing\-derived indicators introduced in Section[2](https://arxiv.org/html/2606.24501#S2), including signals for exclusion notes, alternative candidate forms, and related counts\.

#### Retrieval\-based features\.

We derive retrieval\-based features by treating the task as context\-guided cross\-lingual retrieval of the English target\. We build a fixed English candidate bank from possible target words, encode candidates with mBERT\(Devlinet al\.,[2019](https://arxiv.org/html/2606.24501#bib.bib5)\), and map them into a shared multilingual embedding space\. For each instance, the L1 context and source word are encoded as a source\-side representation and matched against this bank\. Candidates are reranked using the available constraints: initial\-letter clue, target length, and part of speech\. Matching candidates are promoted, while mismatches are downweighted or excluded\. We then extract features describing gold\-target support, including rank before and after reranking, cosine and reranked scores, top\-candidate scores, margins, nearest\-neighbour confidence and entropy, gold\-target probability measures, and the number of valid candidates\. The gold target’s retrieval position is encoded as its rank when retrieved and as 0 otherwise\. These features capture how recoverable the target is from the L1 context and source word\.

#### Masked language model and surprisal features\.

We use the MLM head of mBERT\(Devlinet al\.,[2019](https://arxiv.org/html/2606.24501#bib.bib5)\)to estimate how predictable the English target is in context\. Unlike retrieval features, which compare the target with an external candidate bank, MLM features come from the output distribution at the masked target position\. We apply target constraints, especially initial letter and length, so the scores reflect both contextual fit and target\-form compatibility\. Features include gold\-target log probability and rank, masked\-token entropy, top\-1 indicators, prediction margins, target–prediction embedding similarity, and related confidence measures\.

Because targets may contain multiple wordpieces, probability\-based quantities are computed at the subword level and averaged across the target span\. For a target sequencet1:mt\_\{1:m\}, we use three surprisal variants\. Simultaneous masking replaces the full target span with\[MASK\]:

smasked​\(t1:m\)=−1m​∑j=1mlog⁡P​\(tj∣𝐜mask\)\.s\_\{\\mathrm\{masked\}\}\(t\_\{1:m\}\)=\-\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\log P\\\!\\left\(t\_\{j\}\\mid\\mathbf\{c\}\_\{\\mathrm\{mask\}\}\\right\)\.\(1\)
Pseudo\-log\-likelihood \(PLL\) masks one target wordpiece at a time while keeping the remaining pieces visible\(Salazaret al\.,[2020](https://arxiv.org/html/2606.24501#bib.bib10)\):

sPLL​\(t1:m\)=−1m​∑j=1mlog⁡P​\(tj∣𝐜\(j\)\),s\_\{\\mathrm\{PLL\}\}\(t\_\{1:m\}\)=\-\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\log P\\\!\\left\(t\_\{j\}\\mid\\mathbf\{c\}^\{\(j\)\}\\right\),\(2\)where𝐜\(j\)\\mathbf\{c\}^\{\(j\)\}masks only tokentjt\_\{j\}, allowing each token to be predicted with access to the other target subwords\.

Finally, chain\-rule surprisal scores the target incrementally from left to right:

schain​\(t1:m\)=−1m​∑j=1mlog⁡P​\(tj∣𝐜,t1:j−1\)\.s\_\{\\mathrm\{chain\}\}\(t\_\{1:m\}\)=\-\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\log P\\\!\\left\(t\_\{j\}\\mid\\mathbf\{c\},t\_\{1:j\-1\}\\right\)\.\(3\)These measures capture contextual predictability under full\-span masking, single\-token masking, and sequential prediction\.

#### Semantic features\.

We derive semantic\-domain features from the UCREL Semantic Analysis System \(USAS\), which assigns lexical items to 21 major discourse fields and 232 finer\-grained categories\(Raysonet al\.,[2004](https://arxiv.org/html/2606.24501#bib.bib14)\)\. For each item, we annotate the L1 source word and English target with all available USAS tags\. We include overlap features for identical tags, shared fine\-grained prefixes, and shared major fields\. We also compute weighted and unweighted tag entropy to capture semantic ambiguity, with higher entropy indicating less stable interpretation\.

To capture relatedness beyond exact overlap, we compute a semantic\-shift score\. LetSSandTTbe the USAS tag sets for the L1 source and English target\. For each source–target tag pair, we assign a graded relation score based on the strongest relation: exact tag overlap, shared fine\-grained prefix, shared major field, related major fields, or unrelated fields\. Item\-level soft similarity is computed over the full tag sets, and semantic shift is defined as its complement:

shiftsem​\(S,T\)=1−simsoft​\(S,T\)\.\\mathrm\{shift\}\_\{\\mathrm\{sem\}\}\(S,T\)=1\-\\mathrm\{sim\}\_\{\\mathrm\{soft\}\}\(S,T\)\.\(4\)Lower values indicate stronger semantic alignment; higher values indicate larger semantic\-domain shift\. For example,obra\(‘literary work’ in Spanish\) has the semantic tags ofA1\.1\.1\|I3\.1, which do not overlap withcomedyE4\.1\+\|K4\|Q4\.3\. Still, both refer to related fields, giving soft similarity 0\.20 and semantic shift 0\.80\. Coverage gaps are handled using fixed penalties for unmatchedZ99cases\. Appendix[C](https://arxiv.org/html/2606.24501#A3),Table[7](https://arxiv.org/html/2606.24501#A3.T7)gives further examples with full USAS category names\.

#### Cognate features\.

We derive cognate\-oriented features to capture lexical\-transfer signals between the L1 source word and the English target\. These include orthographic similarity, multilingual embedding cosine similarity, Levenshtein edit\-distance similarity, characternn\-gram overlap, and prefix/suffix overlap\. We also add lexical\-relation features from CogNet, a multilingual cognate database\(Batsurenet al\.,[2019](https://arxiv.org/html/2606.24501#bib.bib15),[2022](https://arxiv.org/html/2606.24501#bib.bib16)\)\. Because the task covers English, Spanish, German, and Chinese, we restrict CogNet to entries involving these languages\. These features are expected to be most informative for Spanish–English and German–English, while also testing whether cognate relations provide useful signals for Chinese–English\.

### 3\.2Model Architectures

#### Textualised Feature Regression\.

The main text fields and selected engineered features are rendered as text, encoded with a multilingual transformer, and passed to a regression head\. This allows feature–context interactions to be modelled through self\-attention\.

#### Multi\-stage Late Fusion\.

Here the available text fields are concatenated and encoded with a frozen multilingual sentence encoder to obtain dense representations𝐡∈ℝd\\mathbf\{h\}\\in\\mathbb\{R\}^\{d\}\. These are used alone or concatenated with engineered features𝐱\\mathbf\{x\}as input to Ridge, GBDT, XGBoost, and SVR regressors\. Their predictions are combined through ensemble\-style aggregation\(van der Laanet al\.,[2007](https://arxiv.org/html/2606.24501#bib.bib19)\), following feature\-based regression approaches in quality estimation and related prediction tasks\(Rios and Sharoff,[2016](https://arxiv.org/html/2606.24501#bib.bib20)\)\.

#### Neural Fusion\.

This architecture jointly learns from contextual text representations and structured tabular features\. For each instance, the input sequenceSiS\_\{i\}concatenatesL1\_context,L1\_source\_word,en\_target\_word, and, in some configurations, an additional target clue using \[SEP\] tokens\.

A multilingual transformerfθf\_\{\\theta\}encodesSiS\_\{i\}, and its token\-level representations are pooled into a text vector:

𝐡i=Pool​\(fθ​\(Si\)\)\.\\mathbf\{h\}\_\{i\}=\\mathrm\{Pool\}\(f\_\{\\theta\}\(S\_\{i\}\)\)\.\(5\)
In parallel, a feed\-forward tabular branch maps the engineered feature vector𝐱i\\mathbf\{x\}\_\{i\}into a dense representation𝐱i′\\mathbf\{x\}^\{\\prime\}\_\{i\}\. The two representations are concatenated:

𝐳i=\[𝐡i;𝐱i′\],\\mathbf\{z\}\_\{i\}=\[\\mathbf\{h\}\_\{i\};\\mathbf\{x\}^\{\\prime\}\_\{i\}\],\(6\)and passed to an MLP to predict the difficulty score:

y^i=MLP​\(𝐳i\)\.\\hat\{y\}\_\{i\}=\\mathrm\{MLP\}\(\\mathbf\{z\}\_\{i\}\)\.\(7\)
Text representations vary by model family: neural fusion uses attention pooling over final hidden states, textualised regression uses the standard\[CLS\]\[\\mathrm\{CLS\}\]representation, and late fusion uses dense sentence embeddings, optionally followed by PCA\.

LanguageSystemRMSE↓\\downarrowΔ\\DeltaRMSE vs Closed↑\\uparrowPearson↑\\uparrowSpearman↑\\uparrowKendall↑\\uparrowesBaseline \(closed\)1\.3570–0\.7480––esLaBSE1\.17320\.18380\.81160\.81360\.6189esXLM\-R\-large1\.11890\.23810\.82910\.83400\.6416esmBERT1\.32690\.03010\.74150\.74590\.5475esmultilingual\-E51\.11340\.24360\.83010\.83350\.6386esBGE\-M31\.09520\.26180\.83240\.83730\.6473esLate fusion1\.26670\.09030\.75130\.75830\.5620esTextualised Feature1\.12610\.23090\.82140\.82390\.6324deBaseline \(closed\)1\.3280–0\.7530––deLaBSE1\.15680\.17120\.80150\.81390\.6208deXLM\-R\-base1\.22110\.10690\.78850\.80580\.6093demBERT1\.26930\.05870\.74810\.76400\.5665demultilingual\-E5\-large1\.08730\.24070\.82340\.84140\.6446deBGE\-M31\.14570\.18230\.81350\.82600\.6284deLate fusion1\.26270\.06530\.72190\.73650\.5389deTextualised Feature1\.09610\.23190\.82520\.84170\.6466cnBaseline \(closed\)1\.1750–0\.7360––cnLaBSE0\.99370\.18130\.81790\.82200\.6343cnXLM\-R\-large1\.2552\-0\.08020\.67040\.66990\.4906cnmBERT1\.12270\.05230\.76730\.77370\.5868cnmultilingual\-E50\.97260\.20240\.83400\.83870\.6493cnBGE\-M30\.96810\.20690\.83510\.84280\.6568cnLate fusion1\.09290\.08210\.76230\.77220\.5775cnTextualised Feature0\.97500\.20000\.82830\.83620\.6459Table 1:Main development\-set results \[dev\] for the closed track\.Δ\\DeltaRMSE is computed asRMSEbaseline−RMSEsystem\\mathrm\{RMSE\}\_\{\\text\{baseline\}\}\-\\mathrm\{RMSE\}\_\{\\text\{system\}\}, so higher values indicate larger improvements\.best,near\-best,close to baseline\.

### 3\.3Experimental Setup and Optimisation

We train separate models for Spanish, German, and Chinese\. For each language, we evaluate multilingual PLMs, including XLM\-R and mBERT\(Conneauet al\.,[2020](https://arxiv.org/html/2606.24501#bib.bib4); Devlinet al\.,[2019](https://arxiv.org/html/2606.24501#bib.bib5)\), and sentence\-oriented multilingual encoders, including LaBSE, Multilingual\-E5, and BGE\-M3\(Fenget al\.,[2022](https://arxiv.org/html/2606.24501#bib.bib6); Wanget al\.,[2024](https://arxiv.org/html/2606.24501#bib.bib7); Chenet al\.,[2024](https://arxiv.org/html/2606.24501#bib.bib9)\)\.

All experiments use five\-fold cross\-validation, with out\-of\-fold predictions used for training\-set evaluation and held\-out embeddings used for late fusion to avoid leakage\. Models are optimised with AdamW and early stopping on validation RMSE, with differential learning rates, warmup, gradient clipping, and mixed precision\. We test both mean squared error and Huber loss, the latter reducing the influence of large residuals; details are given in Appendix Eq\.[8](https://arxiv.org/html/2606.24501#A2.E8)and Appendix[B](https://arxiv.org/html/2606.24501#A2),Table[6](https://arxiv.org/html/2606.24501#A2.T6)\. RMSE is the primary metric used by the shared task organisers, while Pearson, Spearman, and Kendall’sτ\\tauused to assess fit and ranking quality of each method\.

## 4Results

Table[1](https://arxiv.org/html/2606.24501#S3.T1)reports the main development\-set results for the closed track, comparing Textualised Feature Regression, Multi\-stage Late Fusion, and Neural Fusion\. Across all three languages, the best systems substantially outperform the official closed\-track baselines, with RMSE gains of 0\.2618 for Spanish using BGE\-M3, 0\.2407 for German using multilingual\-E5\-large, and 0\.2069 for Chinese using BGE\-M3\.

The results show a clear advantage for sentence\-embedding\-oriented multilingual encoders\. BGE\-M3 and multilingual\-E5\-large, both based on XLM\-RoBERTa\-large, are consistently stronger, while LaBSE, a multilingual BERT\-based sentence embedding model, remains competitive\. This suggests that the main advantage comes not only from the underlying multilingual backbone, but also from training the model specifically for sentence\-level cross\-lingual representation\. In contrast, the standard token encoders are weaker, mBERT is generally less competitive, and XLM\-R is less stable, performing reasonably well for Spanish and German but falling below the baseline for Chinese\.

For the Late Fusion and Textualised Feature Regression settings, we therefore used the two strongest sentence\-embedding encoders, BGE\-M3 and multilingual\-E5\-large, and report the best development\-set result for each language\. The modelling strategy also has a clear effect\. Late Fusion performs worse than both Textualised Feature Regression and Neural Fusion, indicating that simply combining frozen sentence embeddings with classical regressors is not the most effective way to use these representations\. In contrast, Neural Fusion benefits from combining encoder representations with engineered features in a more integrated way\. Since the best development results are obtained with Neural Fusion using BGE\-M3 or multilingual\-E5\-large, we use this setting for the subsequent feature\-ablation and feature\-importance analyses\.

### 4\.1Feature importance and ablations

LanguageFeature settingOOFDevRMSEPearsonSpearmanKendallRMSEPearsonSpearmanKendallesAll features1\.09880\.81720\.81490\.62231\.09520\.83240\.83730\.6473Top\-1/group \(Kendall\)1\.14290\.79310\.79000\.59541\.12560\.82440\.82980\.6378Top\-3/group \(Kendall\)1\.12210\.80610\.80360\.60891\.11430\.83080\.83470\.6452Frequency1\.09870\.81220\.80840\.61571\.08550\.83200\.83500\.6450Surface\-form1\.13430\.80360\.80130\.60761\.17390\.82160\.82380\.6315MLM1\.29280\.72570\.71740\.52741\.10990\.82950\.82970\.6404Surprisal1\.12680\.79970\.79660\.60221\.09730\.83040\.83540\.6440Semantic1\.12760\.80430\.80290\.60991\.12090\.82690\.82750\.6381Cognate1\.12810\.80590\.80360\.61041\.12350\.82370\.82890\.6386Features only1\.32480\.70270\.69270\.50301\.32740\.71360\.70790\.5130deAll features1\.07610\.79870\.80360\.60801\.08730\.82340\.84140\.6446Top\-1/group \(Kendall\)1\.05380\.82100\.82540\.63131\.10560\.82500\.83980\.6428Top\-3/group \(Kendall\)1\.04760\.80960\.81350\.61891\.13060\.81230\.82770\.6304Frequency1\.04520\.81470\.82010\.62561\.09410\.82350\.83810\.6413Surface\-form1\.08050\.80600\.81250\.61681\.10730\.81730\.83350\.6361MLM1\.07780\.80280\.80880\.61291\.12550\.81320\.82800\.6285Surprisal1\.08310\.79630\.80250\.60601\.08610\.82610\.84030\.6454Semantic1\.07440\.79750\.80360\.60751\.13430\.81400\.83000\.6334Cognate1\.08060\.80830\.81330\.61901\.09850\.82250\.83890\.6431Features only1\.24910\.70860\.70800\.51461\.33390\.68250\.68930\.4986cnAll features0\.95940\.82350\.82000\.63000\.96810\.83510\.84280\.6568Top\-1/group0\.99510\.82580\.83650\.64930\.98440\.81390\.81340\.6215Top\-3/group0\.96510\.82780\.82310\.63301\.01260\.82170\.83340\.6456Frequency0\.94920\.83140\.82980\.63970\.98990\.82930\.83550\.6472Surface\-form0\.98570\.81730\.81650\.62450\.98400\.82980\.84190\.6498MLM0\.96490\.82180\.82050\.63050\.97960\.82800\.84030\.6498Surprisal0\.96890\.82250\.82180\.63051\.00890\.82310\.83510\.6460Semantic1\.00530\.80500\.80250\.61021\.00300\.82610\.83350\.6433Cognate0\.96930\.81860\.81640\.62560\.98070\.82640\.83570\.6463Features only1\.16470\.71590\.70800\.51901\.23770\.68230\.69250\.5081

Table 2:Feature ablation results for Spanish, German, and Chinese\. Top\-1/group and Top\-3/group are built by selecting the highest\-ranked one or three features within each feature group according to Kendall correlation with the training GLMM score\.best,near\-best,worst\.LanguageRunRMSE↓\\downarrowPearson↑\\uparrowesAll features1\.1320\.813Frequency1\.1340\.808Top\-3/family1\.1400\.813deAll features1\.0790\.819Frequency1\.0370\.834Surprisal1\.0780\.818cnAll features0\.9300\.856Frequency0\.8910\.860Top\-1/family0\.9190\.858Table 3:Official closed\-track test results for UOL@IDEM submissions\. Green highlights the best run\.We analyse the engineered features in three steps\. First, we compute Kendall’sτ\\taucorrelation between each feature and the goldGLMM scorefor each language\. Second, we use these rankings to construct two compact feature sets: Top\-1 per feature family, which keeps the single highest\-ranked feature in each family, and Top\-3 per feature family, which keeps the three highest\-ranked features in each family\. Third, we compare these compact settings with models trained using only one feature family at a time\. This separates feature–target association from the predictive value of compact subsets and single feature families\.

The Kendall rankings show that frequency and lexical familiarity are the most consistent individual signals\. In Spanish, the top\-ranked features are subtitle\-based target frequency \(\|τ\|=0\.2730\|\\tau\|=0\.2730\), the retrieval prior combining clue/context support and candidate\-space size \(\|τ\|=0\.2643\|\\tau\|=0\.2643\), and subtitle contextual diversity \(\|τ\|=0\.2572\|\\tau\|=0\.2572\)\. In German, the retrieval prior ranks first \(\|τ\|=0\.2985\|\\tau\|=0\.2985\), followed by subtitle frequency \(\|τ\|=0\.2837\|\\tau\|=0\.2837\) and subtitle contextual diversity \(\|τ\|=0\.2680\|\\tau\|=0\.2680\)\. In Chinese, the strongest associations are subtitle frequency \(\|τ\|=0\.4128\|\\tau\|=0\.4128\), subtitle contextual diversity \(\|τ\|=0\.4127\|\\tau\|=0\.4127\), and the KELLY lexical familiarity score \(\|τ\|=0\.4029\|\\tau\|=0\.4029\)\.

Table[2](https://arxiv.org/html/2606.24501#S4.T2)shows that that frequency\-based features \(subtitle frequency and contextual diversity\) emerge as the most consistent and strongest individual predictors across languages, but its role differs by L1\. Spanish is the clearest frequency\-driven case: the frequency\-only model gives the best Dev RMSE, 1\.0855, slightly ahead of all features, 1\.0952, with nearly identical OOF RMSE, 1\.0987 versus 1\.0988\. German shows a more balanced pattern, where surprisal gives the best Dev RMSE, 1\.0861, narrowly ahead of all features, 1\.0873, while frequency, 1\.0941, and cognate/form\-similarity, 1\.0985, remain close\. Chinese has the most distributed signal: all features perform best, with Dev RMSE 0\.9681, but MLM, cognate/form\-similarity, lexical/surface form, and frequency all remain competitive\.

The official test results in Table[3](https://arxiv.org/html/2606.24501#S4.T3)partly confirm this pattern\. The best UOL@IDEM runs achieve RMSE 1\.132 for Spanish, 1\.037 for German, and 0\.891 for Chinese\. Spanish is best with the all\-features run, while German and Chinese are best with the frequency\-oriented run\. Thus, although richer feature combinations remain useful during development, frequency\-based predictors generalise most robustly on the hidden test set for German and Chinese\.

### 4\.2Error analysis

The feature\-importance and ablation results show that in addition to the baseline sentence encoder, the models learn a strong difficulty signal, especially from frequency and lexical familiarity\. However, a strong ranking does not guarantee good calibration\. We therefore examine where frequency becomes insufficient, whether predictions are compressed across the difficulty scale, and which feature patterns distinguish low\-error from high\-error cases\.

We first examine calibration by dividing the development items into five quantile\-based gold\-difficulty bands\. Band 1 contains the easiest items and Band 5 the hardest items\. For each language and band, we compute RMSE, MAE, Kendall’sτ\\tau, and signed bias, defined asy^−y\\hat\{y\}\-y, where positive values indicate overprediction of difficulty\(Kendall,[1938](https://arxiv.org/html/2606.24501#bib.bib30)\)\. Uncertainty is estimated using bootstrap resampling\(Efron and Tibshirani,[1994](https://arxiv.org/html/2606.24501#bib.bib37)\)\.

Figure[1](https://arxiv.org/html/2606.24501#S4.F1)shows that the language\-specific neural fusion models broadly follow the gold difficulty scale, but with a clear compression toward the centre\. This compression is visible in the solid mean\-bias line, where bias is defined as predicted difficulty minus gold difficulty\. In Band 1, which contains the easiest items, the bias is strongly positive:\+1\.21\+1\.21for German,\+1\.28\+1\.28for Spanish, and\+1\.03\+1\.03for Chinese\. This means that genuinely easy items are predicted as more difficult than they are\. At the other end of the scale, Band 5 shows the opposite pattern: the bias is negative, with values of−0\.30\-0\.30,−0\.32\-0\.32, and−0\.27\-0\.27, respectively, indicating that the hardest items are predicted as slightly easier than their gold scores\. These two effects show that predictions are pulled away from the extremes and toward the middle of the difficulty scale\. The same pattern is also reflected in the mean difficulty lines: the predicted mean difficulty line changes less sharply across bands than the gold mean difficulty line\. Thus, the main weakness is not a failure to rank items by difficulty, but a calibration problem in which the model underestimates the full spread of the difficulty scale\. The full error\-analysis materials are available online\.333[https://tinyurl\.com/UoL\-BEA\-Error\-Analysis](https://tinyurl.com/UoL-BEA-Error-Analysis)

Lang\.RMSEMAEBiasτ\\tauBand bias patternde1\.1170\.850\+0\.3460\.629\+1\.21→−0\.30\+1\.21\\rightarrow\-0\.30es1\.1110\.834\+0\.3310\.651\+1\.28→−0\.32\+1\.28\\rightarrow\-0\.32zh0\.9750\.724\+0\.2580\.648\+1\.03→−0\.27\+1\.03\\rightarrow\-0\.27

Table 4:Overall development\-set error profile\. Bias is the mean signed errory^−y\\hat\{y\}\-y\. The final column summarises the change from Band 1 to Band 5, showing scale compression in all three languages\.![Refer to caption](https://arxiv.org/html/2606.24501v1/x1.png)Figure 1:Calibration by gold difficulty band for German, Spanish, and Chinese\. Bands B1–B5 are ordered from easiest to hardest according to the gold difficulty score\. The solid line shows mean prediction bias, defined as predicted difficulty minus gold difficulty, and the shaded region shows bias±\\pmMAE\. Positive bias in the easiest bands indicates over\-prediction of difficulty, while negative bias in the hardest band indicates under\-prediction\.Table[4](https://arxiv.org/html/2606.24501#S4.T4)summarises the same pattern at language level\. Chinese has the lowest RMSE and MAE, but it still shows the same movement from overprediction in the easiest band to mild underprediction in the hardest band\. This connects directly to the ablation results: frequency provides a robust global difficulty signal, but it can also pull predictions toward an average level of difficulty rather than fully adapting to the extremes of the learner\-specific scale\.

![Refer to caption](https://arxiv.org/html/2606.24501v1/x2.png)Figure 2:Matched low\-error versus high\-error contrasts within language and difficulty band\. Positive values indicate features higher in low\-error items; negative values indicate features higher in high\-error items\.Absolute\-error associations with individual features are weak: most Kendall correlations are below\|τ\|=0\.05\|\\tau\|=0\.05\. The strongest cue is prediction entropy,𝚙𝚛𝚎𝚍​\_​𝚎𝚗𝚝\\mathtt\{pred\\\_ent\}, withτ=0\.048\\tau=0\.048, followed by MLM entropy, learner\-frequency percentile, target length, semantic\-domain entropy, prediction margin, and cognate similarity\. Residual errors therefore appear when several cues diverge, such as frequency, contextual predictability, semantic ambiguity, form similarity, and L1–English transparency\.

To examine these interactions more directly, we compare low\-error and high\-error items within the same language and difficulty band\. This controls for broad gold difficulty: we are not simply comparing easy words with hard words, but asking what distinguishes better and worse predictions among items of comparable true difficulty\.

Within each language–band group, we define low\-error items as the lowest\-error quartile and high\-error items as the highest\-error quartile\. Since many features are skewed, ordinal, or tied, we use Mann–Whitney tests and Cliff’sδ\\deltato compare feature distributions, and apply Benjamini–Hochberg correction to control the false discovery rate\(Mann and Whitney,[1947](https://arxiv.org/html/2606.24501#bib.bib31); Cliff,[1993](https://arxiv.org/html/2606.24501#bib.bib39); Benjamini and Hochberg,[1995](https://arxiv.org/html/2606.24501#bib.bib33)\)\. Figure[2](https://arxiv.org/html/2606.24501#S4.F2)reports the standardised contrast between the two groups, computed as low\-error minus high\-error\. Positive values indicate features that are higher in better\-predicted items; negative values indicate features that are higher in worse\-predicted items\.

The controlled contrasts clarify the source of the calibration errors\. Low\-error items are usually those where several cues point in the same direction: frequency, lexical form, semantic evidence, and contextual predictability jointly support the predicted difficulty\. High\-error items arise when these cues conflict, so the model receives mixed evidence about whether the item should be easy or difficult\.

In German, low\-error items show stronger form and contextual\-predictability signals, including target length, source length, target syllable count, MLM rank, and surprisal features\. High\-error items are more associated with subtitle frequency, subtitle contextual diversity, and cognate similarity\. This pattern indicates that German predictions are most stable when frequency is supported by contextual and form\-based evidence, but less stable when frequency or surface similarity dominates without enough contextual support\.

Spanish shows the clearest example of this conflict\. The most error\-prone cell is Spanish Band 1, where genuinely easy items are often overpredicted\. Within this band, the strongest descriptive contrasts involve subword surprisal, MLM log probability, and pseudo\-log\-likelihood surprisal\. The largest contrast is for𝚜𝚞𝚛𝚙​\_​𝚜𝚞𝚋​\_​𝚜𝚞𝚖\\mathtt\{surp\\\_sub\\\_sum\}, with Cliff’sδ=−0\.298\\delta=\-0\.298and a raw Mann–Whitneyp=0\.031p=0\.031; however, this effect does not remain significant after Benjamini–Hochberg correction \(pBH=0\.522p\_\{\\mathrm\{BH\}\}=0\.522\), so we treat it as exploratory\. The pattern is still informative: some easy items may be frequent, familiar, cognate\-supported, or transparent for Spanish\-speaking learners even when the immediate context does not make them highly predictable to an MLM\-style model\. The model then overpredicts difficulty because contextual predictability conflicts with learner\-oriented familiarity\.

Chinese has the strongest overall performance, but it follows the same mechanism\. Low\-error items combine form, semantic\-shift, cognate\-similarity, and probability\-based support, including target length, target syllables, semantic shift, predicted log probability, cognate similarity, and MLM log probability\. High\-error items are more strongly associated with frequency features such as word\-frequency value, subtitle contextual diversity, subtitle log\-frequency, and subtitle frequency\. Thus, even for Chinese, predictions are most reliable when frequency is reinforced by semantic and form\-based evidence rather than operating as the main cue alone\.

Overall, the error analysis refines the feature\-ablation findings\. Frequency is the strongest global cue and explains much of the high ranking performance, but it does not guarantee calibrated predictions\. This explains why high Kendall correlations can coexist with systematic calibration errors: the models often rank items correctly, but their predicted scores are compressed toward the middle, leading to overprediction for easy items and underprediction for hard items\.

## 5Conclusion

This paper presented UOL@IDEM, a closed\-track system for L1\-aware English vocabulary difficulty prediction\. The system combines multilingual contextual representations with engineered features capturing frequency, lexical surface form, surprisal, retrieval evidence, cognate\-like similarity, and semantic alignment\.

Across the development set, neural fusion with sentence\-embedding encoders gives the strongest performance, improving over the closed\-track baselines by 0\.2618 RMSE for Spanish, 0\.2407 for German, and 0\.2069 for Chinese\. On the official test set, the best submissions achieve RMSE scores of 1\.132, 1\.037, and 0\.891 for Spanish, German, and Chinese, respectively\.

The analysis shows that explicit linguistic features remain useful even with strong multilingual encoders\. Frequency is the most stable signal, while form, surprisal, retrieval, cognate, and semantic features provide complementary evidence for L1\-aware prediction\. At the same time, the error analysis shows that strong ranking performance does not always translate into well\-calibrated difficulty estimates, especially at the easiest and hardest ends of the scale\.

Future work should focus on calibration and on more compact models that retain the most informative frequency, form, surprisal, and semantic cues while better modelling L1\-specific transfer and contextual recoverability\.

## 6Limitations

Our work is limited to the closed\-track shared\-task data, so we do not use external learner corpora, larger lexical resources, multilingual joint training, or large language models\. These could be useful for rare targets, noisy source\-word entries, and ambiguous source–target mappings\.

The engineered features improve interpretability but add pipeline complexity, including preprocessing, frequency lookup, retrieval scoring, surprisal estimation, semantic tagging, and cognate/similarity scoring\. Following the closed\-track requirements, we train separate models for each L1, which does not explicitly model shared versus language\-specific difficulty signals\.

The main evaluation limitation is calibration\. The models overpredict the easiest items and slightly underpredict the hardest ones, indicating regression toward the middle of the difficulty scale\. This analysis is restricted to the development set because gold labels for the hidden test set are not available\.

## Ethics Statement

This work uses data released for research under the shared\-task conditions\. The task concerns educational assessment and therefore has potential downstream implications for fairness and learner support\. We do not claim that model predictions should replace psychometric validation or expert judgment; rather, they should be used as supportive signals in item development and educational NLP pipelines\.

## Acknowledgments

We thank the BEA 2026 shared\-task organisers for releasing the dataset and the evaluation framework\. This document is part of a project that has received funding from the European Union’s Horizon Europe research and innovation program under Grant Agreement No\. 101132431 \(iDEM Project\)\. The University of Leeds was funded by UK Research and Innovation \(UKRI\) under the UK government’s Horizon Europe funding guarantee \(Grant Agreement No\. 10103529\)\. The views and opinions expressed in this document are solely those of the author\(s\) and do not necessarily reflect the views of the European Union\. Neither the European Union nor the granting authority can be held responsible for them\.

## References

- CogNet: a large\-scale cognate database\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp\. 3136–3145\.External Links:[Link](https://aclanthology.org/P19-1302/),[Document](https://dx.doi.org/10.18653/v1/P19-1302)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px6.p1.1)\.
- K\. Batsuren, G\. Bella, and F\. Giunchiglia \(2022\)A large and evolving cognate database\.Language Resources and Evaluation56,pp\. 1407–1437\.External Links:[Document](https://dx.doi.org/10.1007/s10579-021-09544-6),[Link](https://doi.org/10.1007/s10579-021-09544-6)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px6.p1.1)\.
- Y\. Benjamini and Y\. Hochberg \(1995\)Controlling the false discovery rate: a practical and powerful approach to multiple testing\.Journal of the Royal Statistical Society: Series B \(Methodological\)57\(1\),pp\. 289–300\.External Links:[Document](https://dx.doi.org/10.1111/j.2517-6161.1995.tb02031.x)Cited by:[§4\.2](https://arxiv.org/html/2606.24501#S4.SS2.p7.1)\.
- British Council \(2026\)BEA 2026 shared task: L1\-aware vocabulary difficulty prediction\.Note:[https://www\.britishcouncil\.org/data\-science\-and\-insights/bea2026st](https://www.britishcouncil.org/data-science-and-insights/bea2026st)Accessed 2026\-04\-04Cited by:[§1](https://arxiv.org/html/2606.24501#S1.p3.3)\.
- M\. Brysbaert and B\. New \(2009\)Moving beyond kučera and francis: a critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english\.Behavior Research Methods41\(4\),pp\. 977–990\.External Links:[Document](https://dx.doi.org/10.3758/BRM.41.4.977),[Link](https://doi.org/10.3758/BRM.41.4.977)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px1.p1.1)\.
- J\. Chen, S\. Xiao, P\. Zhang, K\. Luo, D\. Lian, and Z\. Liu \(2024\)M3\-embedding: multi\-linguality, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 2318–2335\.External Links:[Link](https://aclanthology.org/2024.findings-acl.137/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.137)Cited by:[§1](https://arxiv.org/html/2606.24501#S1.p5.1),[§3\.3](https://arxiv.org/html/2606.24501#S3.SS3.p1.1)\.
- N\. Cliff \(1993\)Dominance statistics: ordinal analyses to answer ordinal questions\.Psychological Bulletin114\(3\),pp\. 494–509\.External Links:[Document](https://dx.doi.org/10.1037/0033-2909.114.3.494)Cited by:[§4\.2](https://arxiv.org/html/2606.24501#S4.SS2.p7.1)\.
- A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov \(2020\)Unsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp\. 8440–8451\.External Links:[Link](https://aclanthology.org/2020.acl-main.747/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747)Cited by:[§3\.3](https://arxiv.org/html/2606.24501#S3.SS3.p1.1)\.
- Council of Europe \(2001\)Common european framework of reference for languages: learning, teaching, assessment\.Cambridge University Press\.External Links:[Link](https://www.coe.int/en/web/common-european-framework-reference-languages/level-descriptions)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px1.p1.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Link](https://aclanthology.org/N19-1423/),[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px4.p1.1),[§3\.3](https://arxiv.org/html/2606.24501#S3.SS3.p1.1)\.
- B\. Efron and R\. J\. Tibshirani \(1994\)An introduction to the bootstrap\.Chapman and Hall/CRC,New York\.External Links:[Document](https://dx.doi.org/10.1201/9780429246593)Cited by:[§4\.2](https://arxiv.org/html/2606.24501#S4.SS2.p2.2)\.
- M\. Felice and L\. Skidmore \(2026\)Findings of the BEA 2026 shared task on vocabulary difficulty prediction for english learners\.InProceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2026\),San Diego, California\.Cited by:[§1](https://arxiv.org/html/2606.24501#S1.p3.3)\.
- F\. Feng, Y\. Yang, D\. Cer, N\. Arivazhagan, and W\. Wang \(2022\)Language\-agnostic BERT sentence embedding\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Dublin, Ireland,pp\. 878–891\.External Links:[Link](https://aclanthology.org/2022.acl-long.62/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.62)Cited by:[§1](https://arxiv.org/html/2606.24501#S1.p5.1),[§3\.3](https://arxiv.org/html/2606.24501#S3.SS3.p1.1)\.
- M\. Honnibal, I\. Montani, S\. Van Landeghem, and A\. Boyd \(2020\)SpaCy: industrial\-strength natural language processing in python\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.1212303),[Link](https://doi.org/10.5281/zenodo.1212303)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px2.p1.1)\.
- M\. G\. Kendall \(1938\)A new measure of rank correlation\.Biometrika30\(1/2\),pp\. 81–93\.External Links:[Document](https://dx.doi.org/10.1093/biomet/30.1-2.81)Cited by:[§4\.2](https://arxiv.org/html/2606.24501#S4.SS2.p2.2)\.
- A\. Kilgarriff, F\. Charalabopoulou, M\. Gavrilidou, J\. B\. Johannessen, S\. Khalil, S\. Johansson Kokkinakis, R\. Lew, S\. Sharoff, R\. Vadlapudi, and E\. Volodina \(2014\)Corpus\-based vocabulary lists for language learners for nine languages\.Language Resources and Evaluation48,pp\. 121–163\.External Links:[Document](https://dx.doi.org/10.1007/s10579-013-9251-2),[Link](https://doi.org/10.1007/s10579-013-9251-2)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px1.p1.1)\.
- H\. B\. Mann and D\. R\. Whitney \(1947\)On a test of whether one of two random variables is stochastically larger than the other\.The Annals of Mathematical Statistics18\(1\),pp\. 50–60\.External Links:[Document](https://dx.doi.org/10.1214/aoms/1177730491)Cited by:[§4\.2](https://arxiv.org/html/2606.24501#S4.SS2.p7.1)\.
- K\. North, M\. Shardlow, M\. Zampieri,et al\.\(2023\)Lexical complexity prediction: an overview\.arXiv preprint arXiv:2303\.04851\.External Links:[Link](https://arxiv.org/abs/2303.04851),[Document](https://dx.doi.org/10.48550/arXiv.2303.04851)Cited by:[§1](https://arxiv.org/html/2606.24501#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px1.p1.1)\.
- A\. Ohuoba, S\. Sharoff, and C\. Walker \(2024\)Quantifying the contribution of MWEs and polysemy in translation errors for English–Igbo MT\.InProceedings of the 25th Annual Conference of the European Association for Machine Translation \(Volume 1\),Sheffield, UK,pp\. 537–547\.External Links:[Link](https://aclanthology.org/2024.eamt-1.43/)Cited by:[§1](https://arxiv.org/html/2606.24501#S1.p1.1)\.
- P\. Rayson, D\. Archer, S\. Piao, and T\. McEnery \(2004\)The ucrel semantic analysis system\.InProceedings of the Workshop on Beyond Named Entity Recognition: Semantic Labelling for NLP Tasks in Association with the 4th International Conference on Language Resources and Evaluation \(LREC 2004\),External Links:[Link](https://ucrel.lancs.ac.uk/people/paul/publications/usas_lrec04ws.pdf)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px5.p1.1)\.
- M\. Rios and S\. Sharoff \(2016\)Language adaptation for extending post\-editing estimates for closely related languages\.The Prague Bulletin of Mathematical Linguistics106,pp\. 5–18\.External Links:[Link](https://ssharoff.github.io/publications/2016-pbml.pdf),[Document](https://dx.doi.org/10.1515/pralin-2016-0013)Cited by:[§3\.2](https://arxiv.org/html/2606.24501#S3.SS2.SSS0.Px2.p1.2)\.
- J\. Salazar, D\. Liang, T\. Q\. Nguyen, and K\. Kirchhoff \(2020\)Masked language model scoring\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp\. 2699–2712\.External Links:[Link](https://aclanthology.org/2020.acl-main.240/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.240)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px4.p4.3)\.
- M\. Shardlow, R\. Evans, and M\. Zampieri \(2021\)SemEval\-2021 task 1: lexical complexity prediction\.InProceedings of the 15th International Workshop on Semantic Evaluation \(SemEval\-2021\),Online,pp\. 1–7\.External Links:[Link](https://aclanthology.org/2021.semeval-1.1/),[Document](https://dx.doi.org/10.18653/v1/2021.semeval-1.1)Cited by:[§1](https://arxiv.org/html/2606.24501#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px1.p1.1)\.
- M\. Shardlow, R\. Evans, and M\. Zampieri \(2022\)Predicting lexical complexity in English texts: the CompLex 2\.0 dataset\.Language Resources and Evaluation56\(4\),pp\. 1037–1064\.External Links:[Document](https://dx.doi.org/10.1007/s10579-022-09588-2),[Link](https://link.springer.com/article/10.1007/s10579-022-09588-2)Cited by:[§1](https://arxiv.org/html/2606.24501#S1.p1.1),[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px1.p1.1)\.
- S\. Sharoff, D\. Goldhahn, and U\. Quasthoff \(2017\)Frequency dictionary: Russian\.Frequency Dictionaries, Vol\.9,pp\. 9–14\.Note:Uwe Quasthoff, Sabine Fiedler, Erla Hallsteindóttir \(editors\)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px1.p1.1)\.
- L\. Skidmore, M\. Felice, and K\. Dunn \(2025\)Transformer architectures for vocabulary test item difficulty prediction\.InProceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications \(BEA 2025\),Vienna, Austria,pp\. 160–174\.External Links:[Link](https://aclanthology.org/2025.bea-1.12/),[Document](https://dx.doi.org/10.18653/v1/2025.bea-1.12)Cited by:[§1](https://arxiv.org/html/2606.24501#S1.p2.1)\.
- R\. Speer, J\. Chin, A\. Lin, S\. Jewett, and L\. Nathan \(2018\)LuminosoInsight/wordfreq: v2\.2\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.1443582),[Link](https://doi.org/10.5281/zenodo.1443582)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px1.p1.1)\.
- M\. J\. van der Laan, E\. C\. Polley, and A\. E\. Hubbard \(2007\)Super learner\.Statistical Applications in Genetics and Molecular Biology6\(1\),pp\. 1–23\.External Links:[Document](https://dx.doi.org/10.2202/1544-6115.1309),[Link](https://ideas.repec.org/a/bpj/sagmbi/v6y2007i1n25.html)Cited by:[§3\.2](https://arxiv.org/html/2606.24501#S3.SS2.SSS0.Px2.p1.2)\.
- W\. J\. B\. van Heuven, P\. Mandera, E\. Keuleers, and M\. Brysbaert \(2014\)SUBTLEX\-uk: a new and improved word frequency database for british english\.The Quarterly Journal of Experimental Psychology67\(6\),pp\. 1176–1190\.External Links:[Document](https://dx.doi.org/10.1080/17470218.2013.850521),[Link](https://doi.org/10.1080/17470218.2013.850521)Cited by:[§3\.1](https://arxiv.org/html/2606.24501#S3.SS1.SSS0.Px1.p1.1)\.
- L\. Wang, N\. Yang, X\. Huang, L\. Yang, R\. Majumder, and F\. Wei \(2024\)Multilingual E5 text embeddings: a technical report\.arXiv preprint arXiv:2402\.05672\.External Links:[Link](https://arxiv.org/abs/2402.05672),[Document](https://dx.doi.org/10.48550/arXiv.2402.05672)Cited by:[§1](https://arxiv.org/html/2606.24501#S1.p5.1),[§3\.3](https://arxiv.org/html/2606.24501#S3.SS3.p1.1)\.

## Appendix AFeature Inventory

This appendix summarises the engineered features used in our experiments\. Feature names follow the column names used in the analysis files and figures\. For readability, features are grouped by functional role rather than presented as a single flat inventory\.TableLABEL:tab:appendix\_engineered\_featuresgives the full engineered\-feature inventory\.

Table 5:Engineered features grouped by functional role\.FeatureDescriptionFrequencyfreq\_enEnglish frequency estimate for the target word\.freq\_en\_pct\_rankPercentile rank of the English target frequency\.sub\_wfSUBTLEX word\-frequency measure\.sub\_lg10wfLog\-scaled SUBTLEX word\-frequency measure\.sub\_cdSUBTLEX contextual\-diversity measure\.sub\_lg10cdLog\-scaled SUBTLEX contextual\-diversity measure\.kelly\_rankRank of the target word in a learner\-oriented lexical frequency resource\.kelly\_pctPercentile rank derived from the Kelly learner\-frequency list\.kelly\_pointsDifficulty\-oriented lexical score from the Kelly list\.wf\_zipfZipf\-scaled word\-frequency score for the English target word\.wf\_valueRaw or transformed corpus\-based word\-frequency value\.Lexical, clue, and preprocessingtgt\_lenNumber of characters in the English target word\.src\_lenNumber of characters in the cleaned L1 source word\.tgt\_syllSyllable count of the English target word\.target\_posPart\-of\-speech information for the English target word\.source\_posPart\-of\-speech information for the L1 source word, where available\.clue\_lenLength of the target clue\.clue\_hidden\_charsNumber of hidden characters in the target clue\.clue\_hidden\_ratioProportion of hidden characters in the target clue\.L1\_source\_word\_rawOriginal unnormalised source\-word field preserved during preprocessing\.L1\_source\_word\_excluded\_wordWord appearing inside an exclusion note in the raw source\-word field\.L1\_source\_word\_has\_excluded\_wordIndicator that the source\-word field contains an exclusion note\.L1\_source\_word\_excluded\_note\_typeType of exclusion note detected during preprocessing\.L1\_source\_word\_has\_alternativeIndicator that multiple source\-word candidates are present\.L1\_source\_word\_alternative\_countNumber of alternative source\-word candidates detected\.L1\_source\_word\_alternativesPreserved list of source\-word alternatives extracted during preprocessing\.Retrievalgold\_rankRank position of the gold English target in the candidate list after retrieval and reranking\.gold\_scoreFinal reranked score assigned to the gold target\.top1\_scoreScore of the highest\-ranked retrieved candidate\.top2\_scoreScore of the second\-ranked retrieved candidate\.margin12Difference between the top\-1 and top\-2 candidate scores\.gold\_cosCosine similarity between the gold target and the retrieved representation\.gold\_probProbability\-like score associated with the gold target under retrieval/reranking\.nn\_pred\_cosCosine similarity associated with the nearest\-neighbour prediction\.nn\_pred\_probProbability\-like confidence score for the nearest\-neighbour prediction\.nn\_entropyEntropy of the retrieved candidate distribution\.cand\_countNumber of valid candidate targets considered after filtering\.retrieval\_target\_priorPrior retrieval score for the target independent of full contextual reranking\.retrieval\_target\_in\_contextContext\-sensitive retrieval score for the target\.baseline\_pred\_matches\_targetIndicator that the retrieval baseline predicts the correct target\.baseline\_clue\_overlapDegree of overlap between the predicted candidate and the target clue\.baseline\_pred\_lenLength\-based compatibility between the retrieved candidate and the target constraint\.MLM and surprisalmlm\_logpMean masked\-language\-model log probability assigned to the gold target wordpieces\.mlm\_rankRank of the gold target under the masked\-language\-model prediction\.mlm\_entEntropy of the masked\-token distribution\.mlm\_top1\_matchIndicator that the top MLM prediction matches the gold target\.pred\_logpLog probability of the top predicted token or predicted target sequence\.pred\_entEntropy associated with the prediction distribution\.pred\_marginMargin between the best and second\-best MLM predictions\.mlm\_target\_pred\_cosCosine similarity between gold\-target and predicted\-token embeddings\.surp\_maskedSurprisal computed from simultaneous masking of all target wordpieces\.surp\_pllSurprisal computed using pseudo\-log\-likelihood with one masked wordpiece at a time\.surp\_chainSurprisal computed sequentially via a chain\-rule decomposition over target subwords\.surp\_sub\_sumSum of surprisal contributions across target subwords\.surp\_sub\_meanMean surprisal across target subwords\.Semantic and domain shiftusas\_domain\_matchIndicator of semantic\-domain compatibility between source and target\.usas\_ent\_unwUnweighted entropy over possible semantic\-domain assignments\.usas\_ent\_wtdWeighted entropy over semantic\-domain assignments\.sem\_shiftSoft semantic\-shift score measuring divergence between the L1 source word and the English target\.semantic\_domain\_overlapDegree of overlap between source\- and target\-side semantic domains\.Cognate and cross\-lingual formlev\_simWeighted Levenshtein\-based similarity between the L1 source word and the English target\.cog\_simCross\-lingual form\-similarity score used as a cognate or transparency cue\.char\_ngram\_simCharacter n\-gram similarity between the L1 source word and the English target, where available\.cognet\_matchIndicator or score derived from external cognate\-link evidence, where available\.
## Appendix BNeural\-fusion model settings

CategorySettingValueInput and modelText fieldsL1\_context,L1\_source\_word,en\_target\_word, optionalen\_target\_clueFusion modelEnhanced neural fusionEncoder backbonesXLM\-R, mBERT, LaBSE, multilingual\-E5, BGE\-M3, and related multilingual encodersText poolingAttention pooling or CLS pooling, depending on the model configurationArchitectureTabular branchFeed\-forward network with configurable hidden layers, dropout, and optional residual connectionsPrediction headMLP regression head over the concatenated text and tabular representationsOptimisationOptimiserAdamWEncoder learning rate2×10−52\\times 10^\{\-5\}Task\-head learning rate10−310^\{\-3\}Weight decay0\.01Warmup ratio0\.1ScheduleLinear warmup and decayGradient clippingMaximum norm 1\.0Mixed precisionUsed when GPU support is availableCheckpoint selectionValidation RMSETraining protocolEpochs5Training batch size16Evaluation batch size16 or 32, depending on the experiment configurationEarly stoppingPatience of 2 epochsEvaluationCross\-validation5\-foldMain metricRMSEOther metricsPearson, Spearman, Kendall’sτ\\tauTable 6:Model, training, and optimisation settings used in the neural\-fusion experiments\.The Huber loss used in some optimisation settings is defined in Eq\.[8](https://arxiv.org/html/2606.24501#A2.E8)\.

Lδ​\(a\)=\{12​a2if​\|a\|≤δ,δ​\(\|a\|−12​δ\)otherwise\.L\_\{\\delta\}\(a\)=\\begin\{cases\}\\frac\{1\}\{2\}a^\{2\}&\\text\{if \}\|a\|\\leq\\delta,\\\\ \\delta\\left\(\|a\|\-\\frac\{1\}\{2\}\\delta\\right\)&\\text\{otherwise\.\}\\end\{cases\}\(8\)

## Appendix CUSAS semantic\-shift examples

ExampleSource USAS tags and categoriesTarget USAS tags and categoriesSoft sim\.Soft shiftOperation: exact tag overlapfirme–solidO4\.1\|O4\.5*General appearance and physical properties; Texture*A5\.1\+\|A5\.3\+\|O1\.1\|O4\.5\|T2\+\+*Evaluation: Good/bad; Evaluation: Accuracy; Substances and materials: Solid; Texture; Time: Beginning and ending*0\.670\.33riesgo–riskA1\.4\|A15\-\|I1\|I2\|I2\.1\.3*Chance/luck; Safety/Danger; Money generally; Business; Business: Generally*A1\.4\|A15\-*Chance/luck; Safety/Danger*0\.780\.22Operation: same fine\-grained prefixoyente–listenerS2\.1\|S2\.2\|S5\|X2\.3*People: Female; People: Male; Groups and affiliation; Learn*S2mf\|X3\.2*People; Sensory: Sound*0\.690\.31noticias–newsQ1\|Q4*Linguistic actions, states and processes; The Media*Q4\.2\|Q4\.3\|T3\-\|X2\.2\+*The Media: Newspapers; The Media: TV, radio and cinema; Time: Old/new/young; Knowledge*0\.600\.40Operation: same major semantic fieldladrillo–brickO1\|O4\.3\|O4\.4*Substances and materials generally; Colour and colour patterns; Shape*H1\|O2\|S8\+*Architecture and kinds of houses/buildings; Objects generally; Helping/hindering*0\.430\.57debatir–discussA6\.1\|G1\.2\|Q1\.1*Comparing: Similar/different; Politics; Linguistic actions and communication*Q2\.1\|Q2\.2*Speech: Communicative; Speech acts*0\.420\.58Operation: related major semantic fieldssueño–dreamB3\|T1\.3*Medicines and medical treatment; Time: Period*A5\.2\-\|X4\.1\|X7\+*Evaluation: True/false; Mental object: Conceptual object; Wanting/planning/choosing*0\.180\.82obra–comedyA1\.1\.1\|I3\.1*General actions, making etc\.; Work and employment: Generally*E4\.1\+\|K4\|Q4\.3*Happy/sad: Happy; Drama, theatre and showbusiness; The Media: TV, radio and cinema*0\.200\.80Operation: unrelated major semantic fieldsfuegos–fireworksL1\|O1\|O4\.6*Life and living things; Substances and materials generally; Temperature*G3\|K1*Warfare, defence and weapons; Entertainment generally*0\.050\.95circulación–trafficO4\.4*Shape*G2\.1\-\|I2\.2\|M1\|M3\|M4\|S1\.1\.2\+*Crime, law and order; Business: Selling; Moving/coming/going; Vehicles and transport on land; Shipping/swimming; Reciprocity*0\.050\.95Operation: coverage gapextensión–spreadZ99*Unmatched*A2\.1\|A6\.3\+\|F1\|N3\.3\|N3\.6\|N5\+\|Q4\.2*Affect: Modify/change; Comparing: Variety; Food; Measurement: Distance; Measurement: Area; Quantities; The Media: Newspapers*–0\.85†Table 7:Spanish examples of the soft USAS semantic\-shift measure using richer source and target tag sets\. Each cell reports the full available USAS tag set, with the corresponding category names underneath\. The operation row describes the strongest source–target tag\-pair relation used to characterise each example group, while the soft similarity and soft shift are computed over the full source and target tag sets\. Therefore, an item can contain an exact overlapping tag but still receive a non\-zero shift score when the remaining tags differ\.†Coverage\-gap scores are penalty values rather than observed semantic\-distance scores\.