求真去伪：僧伽罗语-英语神经机器翻译中病理性幻觉的检测

# Fact over Fiction: Detection of Pathological Hallucinations in Sinhala-to-English Neural Machine Translation
Source: [https://arxiv.org/html/2610.11389](https://arxiv.org/html/2610.11389)
Navam ObeysekaraAffiliation:School of Computing, Informatics Institute of Technology, Sri LankaEmail:[navam\.20211058@iit\.ac\.lk](mailto:)Nevidu JayatillekeAffiliation:Department of Computer Science & Engineering, University of Moratuwa, Sri LankaEmail:[nevidu\.25@cse\.mrt\.ac\.lk](mailto:)

###### Abstract

Neural Machine Translation \(NMT\) models, while capable of producing highly fluent outputs, remain vulnerable to hallucinations, which are translations that are natural yet semantically unrelated to the source\. This vulnerability is acute in low\-resource settings like Sinhala\-to\-English, where weak cross\-lingual alignment leads to hallucinations\. This paper introduces a framework for reference\-free hallucination detection in this language pair\. We present a 45,000\-sample synthetic dataset generated through a probabilistic chain of five linguistically motivated corruption strategies, with a semantic rescue mechanism that uses character\-level similarity to distinguish hallucinations from morphological variants\. We fine\-tunemDeBERTa\-v3for token\-level sequence labelling, reaching a token\-level F1 of0\.841±0\.0010\.841\\pm 0\.001over three seeds on a source\-disjoint test set, and study a three\-signal ensemble integrating neural risk scores, sequence log\-probabilities, and cross\-lingual semantic embeddings \(LaBSE\)\. A source\-ablation control shows that the detector relies on the Sinhala source rather than on surface artefacts of the corruption process: shuffling or removing the source reduces sentence\-level AUROC from 0\.970 to chance\. We benchmark eight NMT systems spanning five model families and find that detector firings vary by an order of magnitude across architectures\.

Keywords:Low\-Resource NLP, Hallucinations Detection, Machine Translation

## 1Introduction

While the Transformer architecture\([Vaswani et al\., 2017](https://arxiv.org/html/2610.11389#bib.bib16)\)has established Neural Machine Translation \(NMT\) as the gold standard for cross\-lingual communication, models remain critically vulnerable to hallucinations\. These are fluent falsehoods that deceive users by mimicking valid output while fabricating information entirely\([Ji et al\., 2023](https://arxiv.org/html/2610.11389#bib.bib11);[Lee et al\., 2018](https://arxiv.org/html/2610.11389#bib.bib13);[Guerreiro et al\., 2023](https://arxiv.org/html/2610.11389#bib.bib19)\)\.

This risk is magnified in Low\-Resource Languages \(LRLs\), where the scarcity of high\-quality parallel corpora leads to model uncertainty and exposure bias\([Wang and Sennrich, 2020](https://arxiv.org/html/2610.11389#bib.bib24)\)\. The reverse translation direction presents unique challenges\. When translating from a low\-resource source \(Sinhala\) to a high\-resource target \(English\), the model’s strong English language model can generate plausible\-sounding text bearing no relation to the original content\. While existing literature has predominantly focused on English\-to\-Sinhala translation\([De Silva, 2026](https://arxiv.org/html/2610.11389#bib.bib22)\), the reverse direction remains severely under\-explored\.

A complicating factor is Sinhala’s severe diglossia\. Real\-world inputs frequently contain colloquial syntax, including sections that routinely omit subjects or pronouns absent from formal training corpora\. Encountering this input, models may resort tosyntactic forcingby inserting grammatically required English agents or auxiliary verbs that have no direct source counterpart\. Currently, existing evaluation benchmarks such as HalOmi\([Dale et al\., 2023b](https://arxiv.org/html/2610.11389#bib.bib12)\)exclude the Sinhala\-English pair entirely, leaving no labelled training data, evaluation framework, or detection baseline\.

To address this, this paper presents a linguistically grounded study on detecting hallucinations in Sinhala\-to\-English NMT, arguing that effective detection in LRLs requires moving beyond standard uncertainty metrics\. We introduce a synthetic dataset of 45,000 sentence pairs generated through five linguistically motivated corruption strategies with a semantic rescue mechanism\. Using this data, we train a reference\-free token\-levelmDeBERTa\-v3detector \(F1 =0\.841±0\.0010\.841\\pm 0\.001over three seeds, on a test set sharing no source sentence with training\)\. Because every synthetic corruption is a monolingual English edit, we add a source\-ablation control that tests whether the detector actually uses the Sinhala source\. We further study a three\-signal ensemble combining neural detection, sequence log\-probability, andLaBSEcross\-lingual semantic similarity, and establish a hallucination benchmark across eight Sinhala\-to\-English NMT systems spanning five model families\.

## 2Related Work

In this section, we review synthetic hallucination data, critiques of its realism, and detection strategies based on confidence, quality estimation, and cross\-lingual embeddings\.

### 2\.1Synthetic Data Approaches

A significant challenge in hallucination detection is the scarcity of naturally occurring labelled errors\.[Zhou et al\. \(2021\)](https://arxiv.org/html/2610.11389#bib.bib20)pioneered the use of synthetic hallucination datasets by framing detection as a sequence labelling task\. Their approach involved corrupting high\-quality source sentences and reconstructing them via a pre\-trainedBARTdenoising autoencoder to produce fluent but semantically incorrect outputs\([Lewis et al\., 2020](https://arxiv.org/html/2610.11389#bib.bib23)\)\. They bridged the bilingual gap with bilingual lexicon induction\([Zhou et al\., 2019](https://arxiv.org/html/2610.11389#bib.bib21)\)and alignment heuristics such asSimAlign\([Jalili Sabet et al\., 2020](https://arxiv.org/html/2610.11389#bib.bib29)\), enabling supervised token\-level detectors such asXLM\-R\([Conneau et al\., 2020](https://arxiv.org/html/2610.11389#bib.bib32)\)where no natural labels existed\.

### 2\.2Natural Hallucinations and Critique

[Guerreiro et al\. \(2023\)](https://arxiv.org/html/2610.11389#bib.bib19)argued that synthetic errors fundamentally differ from actual NMT hallucinations, introducing a taxonomy of oscillatory, strongly detached, and fully detached types\. Their findings revealed that simple model confidence, measured via sequence log\-probability, often performs on par with complex reference\-based metrics while being significantly easier to implement\. Furthermore, they proposed the DEHALLUCINATOR, which attempts to rewrite flagged translations using Monte Carlo dropout\([Gal and Ghahramani, 2016](https://arxiv.org/html/2610.11389#bib.bib14)\)\. This aligns with the hypothesis suggested by[Raunak et al\. \(2021\)](https://arxiv.org/html/2610.11389#bib.bib25), which discusses that hallucinations often stem from model instability or corpus noise rather than fundamental architectural flaws\.

### 2\.3Extended Detection Strategies

The field has explored diverse statistical and glass\-box strategies beyond synthetic methods\.[Martindale et al\. \(2019\)](https://arxiv.org/html/2610.11389#bib.bib28)combined fluency scores with Bag\-of\-Vectors Sentence Similarity to identify translations that are fluent but inadequate, proving semantic similarity more reliable than BLEU\([Papineni et al\., 2002](https://arxiv.org/html/2610.11389#bib.bib27)\)\.[Fomicheva et al\. \(2020\)](https://arxiv.org/html/2610.11389#bib.bib26)evaluate Monte Carlo dropout as an unsupervised QE indicator across six language pairs, with Sinhala\-English placed in their lowest\-resource tier; they report weaker correlation with human judgments for this pair relative to their higher\-resource pairs\([Gal and Ghahramani, 2016](https://arxiv.org/html/2610.11389#bib.bib14);[Kim et al\., 2017](https://arxiv.org/html/2610.11389#bib.bib15)\)\. Parallel to this,COMETestablished a new paradigm for QE by regressing directly on human quality scores\([Rei et al\., 2020](https://arxiv.org/html/2610.11389#bib.bib4)\), whileALTI\+measures the impact on generated tokens, hypothesising that low source contribution is a primary signal for hallucination\([Ferrando et al\., 2022](https://arxiv.org/html/2610.11389#bib.bib18);[Dale et al\., 2023a](https://arxiv.org/html/2610.11389#bib.bib17)\)\.

For cross\-lingual embeddings,[Artetxe and Schwenk \(2019\)](https://arxiv.org/html/2610.11389#bib.bib5)introducedLASER, which[Heffernan et al\. \(2022\)](https://arxiv.org/html/2610.11389#bib.bib3)extended viaLASER3to scale detection to hundreds of languages\. We adoptLaBSE\([Feng et al\., 2022](https://arxiv.org/html/2610.11389#bib.bib31)\), a dual\-encoder architecture optimised with Additive Margin Softmax loss to produce discriminative cross\-lingual sentence embeddings\. Despite these advances, sequence log\-probability is often driven by the target\-language prior rather than faithfulness to the source\. In low\-resource Sinhala\-English settings, where the encoder provides a weak cross\-lingual signal, the decoder can generate a high\-probability English sentence while effectively ignoring the Sinhala source\. This failure mode, characterised by confident but semantically untethered output, is precisely what motivates our reference\-free neural approach\.

A related concern for supervised detectors trained on synthetic data is that they may exploit artefacts of the data\-construction process rather than the intended signal, as shown for hypothesis\-only baselines in natural language inference\([Poliak et al\., 2018](https://arxiv.org/html/2610.11389#bib.bib34);[Gururangan et al\., 2018](https://arxiv.org/html/2610.11389#bib.bib35)\)\. We adopt the corresponding control \(§[4\.2](https://arxiv.org/html/2610.11389#S4.SS2)\)\.

## 3Methodology

The proposed framework111[https://github\.com/Imanjith/synthetic\-sinhala\-english\-dataset](https://github.com/Imanjith/synthetic-sinhala-english-dataset)operates as a two\-stage pipeline consisting of aGenerationstage producing a hypothesis and model confidence score, and aDetectionstage where a fine\-tuned cross\-lingual encoder identifies hallucinated tokens\. Formally, letS=\{s1,…,sm\}S=\\\{s\_\{1\},\\ldots,s\_\{m\}\\\}be a Sinhala source andH=\{h1,…,hn\}H=\\\{h\_\{1\},\\ldots,h\_\{n\}\\\}the English hypothesis\. Hallucination detection learnsf⁡\(S,H\)→Yf\(S,H\)\\rightarrow Y, whereY=\{y1,…,yn\}Y=\\\{y\_\{1\},\\ldots,y\_\{n\}\\\}is a binary label sequence over hypothesis tokens\.

### 3\.1Hallucination Taxonomy

Following[Guerreiro et al\. \(2023\)](https://arxiv.org/html/2610.11389#bib.bib19)and[Raunak et al\. \(2021\)](https://arxiv.org/html/2610.11389#bib.bib25), we partition hallucinations into two categories\.Detached hallucinationsoccur when the model generates content that diverges from the factual meaning of the source on a severity spectrum, ranging from strongly detached outputs to fully detached outputs\. Fully detached outputs have no semantic link whatsoever, whereas strongly detached outputs preserve the surface topic but fabricate specific facts\.Fluent hallucinationsremain locally coherent but introduce subtle semantic errors such as logic inversions and antonym substitutions, whilst remaining grammatically well\-formed\. The mapping from categories to corruption strategies is summarised in Table[1](https://arxiv.org/html/2610.11389#S3.T1)\.

Table 1:Hallucination categories, corruption strategies, and detection challenges\.
### 3\.2Dataset Construction

We construct a synthetic benchmark222[![[Uncaptioned image]](https://arxiv.org/html/2610.11389v1/figures/hf-logo.png)Synthetic\-Sinhala\-English\-Hallucination\-Dataset](https://huggingface.co/datasets/imanjith/synthetic-sinhala-english-hallucination-dataset)for hallucination detection in Sinhala\-to\-English translation\. Starting from a clean parallel corpus, we corrupt English references with five strategies \(alone or chained\), filter the results for grammaticality, and release 45,000 labelled rows\. A balanced 15,000\-row subset, split by source sentence, is used for all detector experiments\.

#### 3\.2\.1Data Source and Preprocessing

The dataset is built on thenllb\-top25k\-ensi\-cleaned333[https://huggingface\.co/datasets/NLPC\-UOM/nllb\-top25k\-ensi\-cleaned](https://huggingface.co/datasets/NLPC-UOM/nllb-top25k-ensi-cleaned)corpus\([Ranathunga et al\., 2024](https://arxiv.org/html/2610.11389#bib.bib9)\), a curated Sinhala\-English parallel dataset of approximately 25,000 sentence pairs\. We removed 4 entries whose English text contained Sinhala characters or no Latin characters \(0\.02%\)\. From the remaining 24,996 pairs, 7,500 source sentences were sampled \(seed 42\), and five negative samples were planned per source, giving7,500×\(1\+5\)=45,0007\{,\}500\\times\(1\+5\)=45\{,\}000rows\. The pipeline is seeded and was run once; the released corpus is used in all experiments\.

#### 3\.2\.2Synthetic Hallucination Generation

Negative samples were generated using a customHallucinationGeneratorapplying five corruption strategies\. Deterministic strategies edit spaCy\([Honnibal and Montani, 2017](https://arxiv.org/html/2610.11389#bib.bib7)\)tokens by index, so that a replacement never hits a different occurrence or a substring of the target word\. A shared entity pool \(PERSON, GPE, ORG, DATE, etc\.\) was pre\-built from 2,000 sampled reference sentences\.

##### Entity Swapping \(NER Swap\)\.

One named entity is replaced with a different entity of the same label from the pre\-built pool\.

##### Semantic Drift\.

NLTK WordNet\([Fellbaum, 1998](https://arxiv.org/html/2610.11389#bib.bib8)\)replaces one non\-entity content word with an antonym of the same part of speech\. Candidates are adjectives and adverbs, or uninflected verbs and nouns \(so that the antonym, a lemma, fits the context\), excludingbe,haveanddo\. Up to three candidates are tried; capitalisation of the original word is preserved\.

##### Syntactic Confusion \(Dependency Swap\)\.

spaCy’s dependency parser identifies the syntactic subject \(nsubj\) and object \(dobj/obj\) of the same verb\. When both are present and neither is a pronoun \(pronoun swaps create trivially detectable case errors\), their positions are swapped to produce subject\-object confusion \(e\.g\., “The manager contacted the client”→\\rightarrow“The client contacted the manager”\)\.

##### Numeric Distortion\.

One whole number \(thousands separators handled, decimals and alphanumeric tokens skipped\) is incremented, decremented, or extended by appending a random digit\. The increment/decrement magnitude is drawn from\[1,max⁡\(10,⌊0\.2​v⌋\)\]\[1,\\max\(10,\\lfloor 0\.2v\\rfloor\)\], wherevvis the original value, ensuring distortion proportional to the original magnitude\.

##### High\-Temperature Sampling\.

English ground\-truth sentences are re\-encoded throughNLLB\-200\-distilled\-1\.3B\([NLLB Team et al\., 2022](https://arxiv.org/html/2610.11389#bib.bib10)\)atT=1\.5T\\\!=\\\!1\.5with nucleus sampling \(top\-p = 0\.95, top\-k = 50\)\. A post\-hoc filter retains only outputs with BERTScore F1<<0\.92\([Zhang et al\., 2020](https://arxiv.org/html/2610.11389#bib.bib30)\)\(roberta\-large, no baseline rescaling\)\. The threshold was set by manual inspection of samples in the 0\.88–0\.94 range: below 0\.92 outputs predominantly altered meaning, while 0\.92–0\.94 mixed genuine changes with close paraphrases\. Of 31,768 sampled outputs, 11,388 \(35\.8%\) were retained\.

#### 3\.2\.3Probabilistic Chain Generation

Rather than applying strategies independently, a probabilistic chain produces compound hallucinations in three phases\.Phase 1 \(Planning\):each negative sample begins as High\-Temperature Sampling \(p = 0\.8\) or a deterministic strategy \(p = 0\.2\)\.Phase 2 \(Batched GPU Generation\):high\-temperature samples are generated in batches of 32 and filtered by BERTScore; rejected samples fall back to a deterministic strategy\.Phase 3 \(Secondary Mutation\):a second, distinct strategy is applied with p = 0\.8 \(e\.g\.,temp\+ner\)\. Whenever a deterministic strategy is needed, all applicable strategies are tried in random order until one changes the text; a row left identical to its reference is relabelled 0\.

Figure 1:Dataset generation pipeline showing the three\-phase probabilistic chain\.
#### 3\.2\.4Grammar Validity Filter

Before balancing, all hallucinated rows are filtered usinglanguage\_tool\_python, ignoring PUNCTUATION, TYPOGRAPHY, WHITESPACE, and CASING categories\. Rows with more than two remaining grammar errors are reverted to the reference and relabelled 0\. The filter rejected 109 of 32,686 hallucinated rows \(0\.33%\)\. It operates purely on grammaticality, so meaning\-altering but syntactically disrupted samples are reverted too\. The effect is small but not uniform across strategies: 78 of the 109 rejections involved High\-Temperature Sampling, which is correspondingly under\-represented after filtering\. We did not manually verify the reverted samples\.

#### 3\.2\.5Corpus Composition and Splits

The released corpus contains 45,000 rows: 32,577 hallucinated \(label 1\) and 12,423 faithful \(label 0\)\. The faithful rows comprise the 7,500 references and 4,923 reverted rows \(4,811 rejected high\-temperature samples with no applicable fallback, 109 grammar rejections, 3 failed fallbacks\), which duplicate their references and are retained, marked by theirmethodfield, for transparency\. The distribution by base \(first\-applied\) strategy is given in Table[2](https://arxiv.org/html/2610.11389#S3.T2)where 13,987 hallucinated rows \(42\.9%\) are compound\.

For detector experiments we draw a balanced subset of 15,000 rows: the 7,500 references as label 0 and 7,500 randomly sampled hallucinated rows as label 1 \(seed 42\)\. This subset is split 70/10/20 by*source sentence*, so that no Sinhala source appears in more than one partition, giving 10,486 training, 1,497 validation and 3,017 test rows \(5,197 / 743 / 1,485 unique sources\)\. Validation is used for checkpoint selection and threshold calibration; the test set is used only for final reporting\.

Table 2:Corpus composition after grammar filtering, by base \(first\-applied\) strategy; compound rows are counted under their first strategy\.

### 3\.3Token\-Level Label Construction

Token\-level binary labels are derived fromdifflib\.SequenceMatcheralignment between corrupted hypothesis and ground\-truth reference\. After light normalisation, the aligner assigns label 0 toequalanddeleteoperations, and candidate label 1 toreplaceandinsertoperations\. Omissions therefore carry no label\.

##### Semantic Rescue for Morphological Variants\.

High\-temperature outputs frequently contain surface variants of reference words \(e\.g\., “colour” vs\. “color”, “believes” vs\. “believed”\) that should not count as hallucinations\. For rows involving High\-Temperature Sampling, each candidate label\-1 wordwhw\_\{h\}is compared against the wordsAAof the aligned reference span \(extended by three words on either side\) using the Ratcliff/Obershelp similarity implemented bydifflib:

sim​\(wh,wr\)=2​M\|wh\|\+\|wr\|,\\text\{sim\}\(w\_\{h\},w\_\{r\}\)=\\frac\{2M\}\{\|w\_\{h\}\|\+\|w\_\{r\}\|\},\(1\)whereMMis the number of matching characters found by recursive longest\-common\-substring matching on lower\-cased words\. Ifmaxwr∈A⁡sim​\(wh,wr\)≥θr=0\.80\\max\_\{w\_\{r\}\\in A\}\\text\{sim\}\(w\_\{h\},w\_\{r\}\)\\geq\\theta\_\{r\}=0\.80, the label is downgraded to 0 \(e\.g\., “colour”/“color”: 0\.91\)\. Rows produced only by deterministic strategies are hallucinations by construction and receive no rescue\. Restricting comparison to the aligned span prevents words that merely occur elsewhere in the reference, such as the two arguments of a dependency swap, from being rescued\. Word\-level labels are propagated to all sub\-word tokens of the word, while source and special tokens receive label−100\-100\.

Rescue does not detect lexical paraphrases between surface\-dissimilar words \(e\.g, “stated” vs\. “said”, similarity 0\.60; “perception” vs\. “understanding”\)\. Such words receive label 1 in high\-temperature rows even when meaning is preserved, which is a source of label noise and of false positives at deployment\.

Sensitivity toθr\\theta\_\{r\}\.In the 2,583 high\-temperature rows of the balanced subset, alignment yields 32,889 candidate label\-1 words\. The share rescued is 7\.7% atθr=1\.0\\theta\_\{r\}=1\.0, 8\.7% at 0\.90, 13\.3% at 0\.80 and 15\.6% at 0\.70\. Of 20 random rescued words per band, all 20 in \[0\.90, 1\.0\) were surface variants, 11 in \[0\.80, 0\.90\), and only 3 in \[0\.70, 0\.80\), where the rest were different words \(e\.g\., “their”/“the”\)\. We keepθr=0\.80\\theta\_\{r\}=0\.80as the point below which rescues are mostly errors, with short words the main source of false rescues\. This analysis concerns the labels only; we did not retrain the detector at other thresholds\.

### 3\.4Translation and Confidence Scoring

The sequence\-level confidence of a generated translation is its log\-probability under the generating model,

LogProb​\(H\|S\)=∑t=1\|H\|log⁡p⁡\(ht∣h<t,S\),\\text\{LogProb\}\(H\|S\)=\\sum\_\{t=1\}^\{\|H\|\}\\log p\(h\_\{t\}\\mid h\_\{<t\},S\),\(2\)summed over the transition scores of the beam\-selected hypothesis as returned by beam search, so that the score is exactly the one the decoder used to selectHH\(we verified that per\-token averages match the reported beam scores for all eight systems\)\. For the synthetic evaluation in §[4\.3](https://arxiv.org/html/2610.11389#S4.SS3), where the hypothesis is given rather than generated, the same quantity is computed by teacher forcing underNLLB\-200\-distilled\-1\.3B\. Scores are min\-max normalised per model \(§[3\.6](https://arxiv.org/html/2610.11389#S3.SS6)\)\.

### 3\.5Hallucination Detection Architecture

mDeBERTa\-v3\(mdeberta\-v3\-base\)\([He et al\., 2023](https://arxiv.org/html/2610.11389#bib.bib6)\)is selected as the primary architecture for two principled reasons\. First, its Replaced Token Detection \(RTD\) pre\-training is structurally identical to our task\. The model learns to classify each token as original or replaced, which is a discriminative objective that makes it≈\{\\approx\}6\.7×\\timesmore sample\-efficient than masked language modelling and whose training signal directly mirrors the hallucination detection setting\. Second, its disentangled attention maintains separate matrices for content and relative position, which is advantageous for the cross\-encoder input formatX=\[CLS\]⊕S⊕\[SEP\]⊕H⊕\[SEP\]X=\\texttt\{\[CLS\]\}\\oplus S\\oplus\\texttt\{\[SEP\]\}\\oplus H\\oplus\\texttt\{\[SEP\]\}\. In the concatenated source\-hypothesis sequence, positional proximity between a Sinhala source token and an English hypothesis token carries no semantic information, as the two segments are separated by a boundary and have different word\-order typologies\. Disentangled attention allows content\-to\-content interactions to be weighted independently of this positional distance\. On XNLI zero\-shot evaluation,mDeBERTa\-v3outperformsXLM\-RoBERTaby 3\.6%\([He et al\., 2023](https://arxiv.org/html/2610.11389#bib.bib6)\), which serves as our baseline\.

#### 3\.5\.1Input Representation and Training

The hypothesis is tokenised word by word so that labels align with words; the source is truncated first if the pair exceeds 512 tokens\. Each hypothesis token hidden statehih\_\{i\}\(d=768d=768\) is classified by a two\-class softmax head,

P⁡\(yi∣X\)=softmax​\(W​hi\+b\),P\(y\_\{i\}\\mid X\)=\\text\{softmax\}\(Wh\_\{i\}\+b\),\(3\)optimised with cross\-entropy\. Configuration:l​r=2×10−5lr=2\\times 10^\{\-5\}, 6 epochs, batch size 8, weight decay 0\.01, fp16, single NVIDIA T4\. The checkpoint with the highest validation token F1 is kept\. Each detector is trained with three seeds \(42, 43, 44\); the same code, data and hyperparameters are used for theXLM\-RoBERTabaseline\.

#### 3\.5\.2Inference and Risk Scoring

A token is flagged ifP⁡\(yi=1∣X\)\>0\.5P\(y\_\{i\}=1\\mid X\)\>0\.5\. TheHallucination Ratioaggregates these into a sentence\-level score:

RH=∑i=1n𝕀⁡\(y^i=1\)nR\_\{H\}=\\frac\{\\sum\_\{i=1\}^\{n\}\\mathbb\{I\}\(\\hat\{y\}\_\{i\}=1\)\}\{n\}\(4\)and a sentence is flagged ifRH\>τR\_\{H\}\>\\tau\. We calibrateτ\\tauon the validation set by sentence\-level F1 overτ∈\{0,0\.01,…,0\.04,0\.05,0\.10,…,0\.50\}\\tau\\in\\\{0,0\.01,\\ldots,0\.04,0\.05,0\.10,\\ldots,0\.50\\\}\. All three seeds selectτ=0\\tau=0\(validation F1 0\.950–0\.961\), so the calibrated rule flags a sentence whenever*any*hypothesis token is flagged\. This removes the insensitivity of ratio thresholds to single\-token hallucinations in long sentences, which matters for domains such as legal or medical translation where a single wrong number or entity is consequential\. We also report the maximum token probability as an alternative, threshold\-free sentence score\.

### 3\.6Three\-Signal Ensemble Framework

The neural detectorRHR\_\{H\}, inverted log\-probabilityC^\\hat\{C\}, andLaBSEsimilarityρ^\\hat\{\\rho\}capture different aspects of translation quality\. Spearman correlations between log\-probability andRHR\_\{H\}range fromρ=−0\.103\\rho=\-0\.103toρ=−0\.437\\rho=\-0\.437across the eight benchmarked systems \(Table[7](https://arxiv.org/html/2610.11389#S4.T7)\), confirming the signals are related but non\-redundant\. Their ensemble score is:

E=wdet⋅RH\+wlp⋅C^\+wsem⋅\(1−ρ^\)E=w\_\{\\text\{det\}\}\\cdot R\_\{H\}\+w\_\{\\text\{lp\}\}\\cdot\\hat\{C\}\+w\_\{\\text\{sem\}\}\\cdot\(1\-\\hat\{\\rho\}\)\(5\)and a sentence is flagged ifEEexceeds a threshold\. We evaluate the originally proposed weighting \(wdet=0\.5w\_\{\\text\{det\}\}=0\.5,wlp=0\.3w\_\{\\text\{lp\}\}=0\.3,wsem=0\.2w\_\{\\text\{sem\}\}=0\.2\) alongside four alternative configurations and a weighting tuned on validation; each configuration’s threshold is tuned on validation \(§[4\.3](https://arxiv.org/html/2610.11389#S4.SS3)\)\.

##### LaBSESemantic Similarity\.

LaBSE\([Feng et al\., 2022](https://arxiv.org/html/2610.11389#bib.bib31)\)encodes source and hypothesis via mean\-pooled, L2\-normalised representations in a shared multilingual space\. Cosine similarity below 0\.80 is treated as semantic divergence, independent of reference translations\.

##### Log\-Probability Normalisation\.

C^=min⁡\(Craw,0\)−CmaxCmin−Cmax\\hat\{C\}=\\frac\{\\min\(C\_\{\\text\{raw\}\},0\)\-C\_\{\\text\{max\}\}\}\{C\_\{\\text\{min\}\}\-C\_\{\\text\{max\}\}\}\(6\)where higherC^\\hat\{C\}indicates lower confidence andCmin,CmaxC\_\{\\text\{min\}\},C\_\{\\text\{max\}\}are taken per model \(on the benchmark\) or from the validation set \(on the synthetic evaluation\)\.

### 3\.7Evaluation Protocol

##### Detector\.

Token\-level precision, recall and F1 over hypothesis tokens of the source\-disjoint test split, reported as mean±\\pmstandard deviation over three seeds, overall and by base strategy\.

##### Source\-ablation control\.

Because every corruption is a monolingual English edit of the reference, a detector could learn to recognise perturbed English without consulting the source\. We therefore evaluate the trained detectors with \(i\) the original source, \(ii\) the source replaced by a different, length\-matched test source, and \(iii\) the source removed, keeping the gold labels fixed; \(iv\) retrain the detector with the source removed from training, validation and test \(a hypothesis\-only baseline\); and \(v\) pair*faithful*test hypotheses with a mismatched source\. Case \(v\) creates fully detached pairs whose English is untouched: a source\-aware detector must flag them, whereas a detector relying on perturbation artefacts will not\.

##### Ensemble\.

On the synthetic validation and test splits, where the gold label refers to the scored hypothesis, we computeRHR\_\{H\}\(seed\-42 detector\), teacher\-forced log\-probability andLaBSEsimilarity\. Weights \(simplex grid, step 0\.1\) and thresholds are tuned on validation and reported on test, together with per\-signal AUROC\.

##### NMT benchmark\.

Eight Sinhala\-to\-English systems translate the 1,485 unique test sources:NLLB\-200\(distilled 600M and 1\.3B\)\([NLLB Team et al\., 2022](https://arxiv.org/html/2610.11389#bib.bib10)\),M2M\-100\(418M and 1\.2B\)\([Fan et al\., 2021](https://arxiv.org/html/2610.11389#bib.bib2)\),mBART\-50\(many\-to\-many and many\-to\-one\)\([Liu et al\., 2020](https://arxiv.org/html/2610.11389#bib.bib33)\), a community fine\-tune ofmT5for Sinhala\-English \(mt5\-sinhalese\-english444[https://huggingface\.co/thilina/mt5\-sinhalese\-english](https://huggingface.co/thilina/mt5-sinhalese-english)\)\([Xue et al\., 2021](https://arxiv.org/html/2610.11389#bib.bib1)\), andHelsinki\-NLP OPUS\-MT\(mul\-en\)\. All use beam search withnum\_beams= 4 andmax\_new\_tokens= 256; all translations completed successfully\. We record which signals fire per sentence and the Spearman correlation between log\-probability andRHR\_\{H\}\.

## 4Evaluation

We first evaluate the detector on the synthetic test split, then verify with a source\-ablation control that it relies on the source sentence\. Next we assess the ensemble and apply it to benchmark eight NMT systems on 1,485 held\-out test sources, and finally examine sources of false positives\.

### 4\.1Detector Performance

mDeBERTa\-v3achieves F1 of0\.841±0\.0010\.841\\pm 0\.001, outperformingXLM\-RoBERTaby 5\.2 points\. The gap is largest in recall \(\+7\.5 points\), consistent with RTD pre\-training, which trains the model to identify substituted tokens\([He et al\., 2023](https://arxiv.org/html/2610.11389#bib.bib6)\)\. Token\-level test performance is presented in Table[3](https://arxiv.org/html/2610.11389#S4.T3)\.

Table 3:Token\-level detector performance on the source\-disjoint test split \(3,017 sentences, hypothesis tokens only\), mean±\\pmstd over three seeds\.Per\-strategy F1 showsmDeBERTa’s advantage is consistent across all strategies\. Dependency Swap is the hardest type \(mDeBERTa: 0\.804 vs\.XLM\-R: 0\.752\), as the swapped words are all present in the source and only their roles change; High\-Temperature Sampling follows \(0\.828 vs\. 0\.784\), reflecting its paraphrase\-related label noise\. Semantic Drift \(0\.934 vs\. 0\.857\), Numeric Distortion \(0\.933 vs\. 0\.855\) and Entity Swap \(0\.916 vs\. 0\.860\) are easiest, as single\-word substitutions contradict an identifiable source counterpart\.

### 4\.2Source\-Ablation Control

Replacing or removing the source at test time reduces token F1 from 0\.841 to about 0\.30 and sentence AUROC from 0\.970 to chance \(0\.54 and 0\.51\), and the detector then flags every test sentence\. Faithful hypotheses are flagged 4\.5% of the time with their own source but 100% with a mismatched one, even though their English is unchanged\. The detector therefore depends on the source\-hypothesis relationship, not the hypothesis alone\. The control results are reported in Table[4](https://arxiv.org/html/2610.11389#S4.T4)\.

The hypothesis\-only baseline reaches token F1 0\.622 and AUROC 0\.850, showing that the synthetic corruptions do leave monolingual cues \(e\.g\., antonym substitutions and disfluent high\-temperature outputs\)\. The full model exceeds it by 0\.22 F1 and 0\.12 AUROC, and can also catch fluent but fully detached translations\.

Table 4:Source\-ablation control on the test split \(mean±\\pmstd over three seeds; AUROC ofRHR\_\{H\}\)\. Gold labels are unchanged across conditions\. Faithful hypotheses are flagged 4\.5%±\\pm0\.9 of the time with their own source and 100% with a mismatched source\.
### 4\.3Ensemble Evaluation

As single signals, the detector \(RHR\_\{H\}\) reaches AUROC 0\.972, well above inverted log\-probability \(0\.880\) andLaBSE\(0\.882\)\. The detector alone attains F1 0\.963 at its validation\-tuned threshold\. The originally proposed weighting \(Det\-Mild\) is lower \(0\.927\), and configurations emphasising log\-probability orLaBSEare lower still \(0\.871–0\.907\)\. The validation\-tuned weighting assigns zero weight to log\-probability and 0\.1 toLaBSE; it matches the detector’s F1 and slightly improves ranking \(AUROC 0\.979\)\. On synthetic data, therefore, the detector carries nearly all of the signal,LaBSEadds a small complementary gain, and log\-probability adds none\. Ensemble results on the synthetic test split are reported in Table[5](https://arxiv.org/html/2610.11389#S4.T5)\.

Table 5:Ensemble configurations on the synthetic test split \(seed\-42 detector; 50\.2% hallucinated\)\. Weights arewd​e​t/wl​p/ws​e​mw\_\{det\}/w\_\{lp\}/w\_\{sem\}; weights and thresholds tuned on validation\.
### 4\.4NMT Benchmarking Results

Each system’s 1,485 translations are partitioned by which of the three signalsfire\(Table[6](https://arxiv.org/html/2610.11389#S4.T6)\): the detector fires when any hypothesis token is flagged \(τ=0\\tau=0\), log\-probability fires when a sentence falls below the model’s own 30th percentile, andLaBSEfires when cosine similarity falls below 0\.80\.

Table 6:NMT benchmarking on 1,485 unique test sources per system\. Partition columns \(All 3 to None\) sum to 1,485;Det firesis the total number of detector\-flagged sentences;LP/wordis mean sequence log\-probability per output word\.##### Detection rates vary by architecture\.

These counts are detector firings rather than verified hallucinations \(§[4\.5](https://arxiv.org/html/2610.11389#S4.SS5)\), so they indicate relative risk rather than error rates\. Detector firings vary by nearly an order of magnitude, from 51 \(NLLB 600M\) to 433 \(OPUS\-MT\)\.OPUS\-MTand themT5fine\-tune produce the most detections and the most all\-three joint detections \(134 and 109\), and they are also among the least confident systems per word\. TheNLLBmodels produce the fewest detections and the highest per\-word confidence, consistent with their strong reported FLORES\-200 performance on Sinhala\([NLLB Team et al\., 2022](https://arxiv.org/html/2610.11389#bib.bib10)\)\. ScalingM2M\-100from 418M to 1\.2B roughly halves detections \(253 to 129\), whereas the twoNLLBcapacities perform similarly \(51 and 70\), and the many\-to\-onemBART\-50variant produces fewer detections than the many\-to\-many variant \(93 vs\. 140\)\. These patterns suggest family and training data matter more than parameter count here, though confirming this requires controlled experiments\.

##### No single signal is sufficient\.

Det Only sentences \(22–97 per system\) are detector flags with normal confidence and highLaBSEsimilarity, which a confidence\-only system would miss\. Log\-probability alone flags about 30% of outputs by construction, producing 183–412 LP\-Only sentences, most of which are uncertain but not flagged by the other signals\.LaBSE\-only detections are 12–67 per system, consistent with a confirmatory role\.

##### Signal independence\.

All are negative and significant \(p<0\.001p<0\.001\)\. Agreement is strongest for the weakest systems \(mT5−0\.437\-0\.437,OPUS\-MT−0\.369\-0\.369\) and weakest forNLLB\(−0\.103\-0\.103,−0\.132\-0\.132\): for strong systems, the detections that remain are largely ones the model itself is confident about\. Spearman correlations between log\-probability andRHR\_\{H\}are reported in Table[7](https://arxiv.org/html/2610.11389#S4.T7)\. Even the strongest correlation accounts for only≈\{\\approx\}19% of the variance in rank, confirming non\-redundancy\.

Table 7:Spearmanρ\\rhobetween log\-probability andRHR\_\{H\}on the benchmark \(allp<0\.001p<0\.001\)\.
##### Threshold sensitivity\.

Validation F1 is stable forτ≤0\.03\\tau\\leq 0\.03\(0\.960–0\.961\) and declines monotonically thereafter \(0\.931 at 0\.05, 0\.571 at 0\.20, 0\.377 at 0\.50\), so ratio thresholds such as the commonly used 0\.20 substantially under\-flag\. On the benchmark, detector firings decrease monotonically withτ\\taufor every system, andOPUS\-MTandmT5remain the two highest\-flagging systems at every threshold tested; ordering among the low\-count systems varies\.

### 4\.5Sources of False Positives

Without human annotation of the benchmark outputs we cannot measure precision on natural translations\. As an indication, of the 129M2M 1\.2Bsentences flagged by the detector, 36 \(27\.9%\) also fall below theLaBSEthreshold; the remainder are either fluent hallucinations that preserve overall sentence meaning or false positives\. Inspection suggests two recurring sources of false positives\.

##### Lexical paraphrasing\.

The detector does not reliably accept synonymous substitutions\. This follows from the label construction \(§[3\.3](https://arxiv.org/html/2610.11389#S3.SS3)\): surface\-dissimilar synonyms such as “stated”/“said” are labelled as hallucinations in high\-temperature training rows, and semantic rescue cannot recover them\.

##### Structural variations\.

When colloquial Sinhala input is encountered, NMT systems insert grammatically required English agents or auxiliaries with no direct source counterpart \(syntactic forcing\)\. Word\-level alignment against the reference penalises such insertions during training, so the detector can flag valid insertions as target\-side hallucinations\.

## 5Conclusion

This work addresses hallucination detection in Sinhala\-to\-English NMT, a task with no prior labelled data, benchmarks, or baselines\. We presented a 45,000\-sample synthetic corpus, a reference\-free token\-level detector \(mDeBERTa\-v3, F1 =0\.841±0\.0010\.841\\pm 0\.001on a source\-disjoint test set\), a source\-ablation control showing that the detector depends on the Sinhala source, a three\-signal ensemble evaluation, and a hallucination benchmark across eight NMT systems\. Detector firings vary by an order of magnitude across architectures, and Det Only detections show that fluent, confident hallucinations exist that log\-probability alone misses\. On synthetic data the detector carries nearly all of the ensemble’s signal, withLaBSEadding a small ranking gain\.

Future work should prioritise human annotation of natural translation errors to measure real precision, and replacedifflibsupervision with cross\-lingual word alignment to reduce paraphrase\-induced label noise\. Ensemble weights and thresholds should also be recalibrated on annotated natural translations, since values tuned on balanced synthetic data do not transfer to the much lower hallucination rates of real output\.

## Limitations

The detector is trained and evaluated on synthetically corrupted data; generalisation to naturally occurring translation errors remains unverified\. All corruptions are monolingual English edits of the reference, and High\-Temperature Sampling re\-encodes English rather than translating from Sinhala, so authentic Sinhala\-origin failure modes may be under\-represented\. The source\-ablation control \(§[4\.2](https://arxiv.org/html/2610.11389#S4.SS2)\) shows that the detector relies on the source, but the hypothesis\-only baseline \(F1 0\.622\) shows that the data also contain exploitable monolingual cues\.

Token labels come fromdifflibalignment, which cannot recognise lexical paraphrases and ignores omissions\. The rescue thresholdθr\\theta\_\{r\}was assessed through its effect on labels \(§[3\.3](https://arxiv.org/html/2610.11389#S3.SS3)\) rather than by retraining at alternative thresholds, and atθr=0\.80\\theta\_\{r\}=0\.80some short, different words are wrongly rescued\. The high\-temperature BERTScore threshold was chosen by qualitative inspection, so paraphrases below it are labelled as hallucinations\. The grammar filter reverts samples on grammaticality alone and disproportionately affects high\-temperature samples \(78 of 109 rejections\)\.

Detector results are averaged over three seeds, but the ensemble evaluation and NMT benchmark use a single detector \(seed 42\)\. The benchmark reports signal\-firing counts without human ground truth, and theLaBSEagreement in §[4\.5](https://arxiv.org/html/2610.11389#S4.SS5)is not a precision estimate\. Because thresholds were calibrated on a balanced synthetic split, they over\-flag real outputs, which is why Table[6](https://arxiv.org/html/2610.11389#S4.T6)reports independent signal firings rather than ensemble decisions\.

## References

- M\. Artetxe and H\. SchwenkMassively multilingual sentence embeddings for zero\-shot cross\-lingual transfer and beyond\.Transactions of the Association for Computational Linguistics7,pp\. 597–610\.External Links:[Link](https://aclanthology.org/Q19-1038/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00288)Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p2.1)\.
- Conneauet al\.\(2020\)A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. StoyanovUnsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 8440–8451\.External Links:[Link](https://aclanthology.org/2020.acl-main.747/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.747)Cited by:[§2\.1](https://arxiv.org/html/2610.11389#S2.SS1.p1.1)\.
- Daleet al\.\(2023a\)D\. Dale, E\. Voita, L\. Barrault, and M\. R\. Costa\-jussàDetecting and mitigating hallucinations in machine translation: model internal workings alone do well, sentence similarity even better\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 36–50\.External Links:[Link](https://aclanthology.org/2023.acl-long.3/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.3)Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p1.1)\.
- Daleet al\.\(2023b\)D\. Dale, E\. Voita, J\. Lam, P\. Hansanti, C\. Ropers, E\. Kalbassi, C\. Gao, L\. Barrault, and M\. R\. Costa\-jussàHalOmi: a manually annotated benchmark for multilingual hallucination and omission detection in machine translation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 638–653\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.42/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.42)Cited by:[§1](https://arxiv.org/html/2610.11389#S1.p3.1)\.
- De Silva \(2026\)N\. De SilvaSurvey on publicly available Sinhala natural language processing tools and research\.arXiv preprint arXiv:1906\.02358\.Cited by:[§1](https://arxiv.org/html/2610.11389#S1.p2.1)\.
- Fanet al\.\(2021\)A\. Fan, S\. Bhosale, H\. Schwenk, Z\. Ma, A\. El\-Kishky, S\. Goyal, M\. Baines, O\. Celebi, G\. Wenzek, V\. Chaudhary,et al\.Beyond English\-centric multilingual machine translation\.Journal of Machine Learning Research22\(107\),pp\. 1–48\.Cited by:[§3\.7](https://arxiv.org/html/2610.11389#S3.SS7.SSS0.Px4.p1.1)\.
- C\. Fellbaum \(Ed\.\) \(1998\)C\. Fellbaum \(Ed\.\)WordNet: an electronic lexical database\.MIT Press\.Cited by:[§3\.2\.2](https://arxiv.org/html/2610.11389#S3.SS2.SSS2.Px2.p1.1)\.
- Fenget al\.\(2022\)F\. Feng, Y\. Yang, D\. Cer, N\. Arivazhagan, and W\. WangLanguage\-agnostic BERT sentence embedding\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 878–891\.External Links:[Link](https://aclanthology.org/2022.acl-long.62/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.62)Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p2.1),[§3\.6](https://arxiv.org/html/2610.11389#S3.SS6.SSS0.Px1.p1.1)\.
- Ferrandoet al\.\(2022\)J\. Ferrando, G\. I\. Gállego, B\. Alastruey, C\. Escolano, and M\. R\. Costa\-jussàTowards opening the black box of neural machine translation: source and target interpretations of the transformer\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 8756–8769\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.599/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.599)Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p1.1)\.
- Fomichevaet al\.\(2020\)M\. Fomicheva, S\. Sun, L\. Yankovskaya, F\. Blain, F\. Guzmán, M\. Fishel, N\. Aletras, V\. Chaudhary, and L\. SpeciaUnsupervised quality estimation for neural machine translation\.Transactions of the Association for Computational Linguistics8,pp\. 539–555\.Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p1.1)\.
- Gal and Ghahramani \(2016\)Y\. Gal and Z\. GhahramaniDropout as a Bayesian approximation: representing model uncertainty in deep learning\.InProceedings of the 33rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.48,pp\. 1050–1059\.Cited by:[§2\.2](https://arxiv.org/html/2610.11389#S2.SS2.p1.1),[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p1.1)\.
- Guerreiroet al\.\(2023\)N\. M\. Guerreiro, E\. Voita, and A\. MartinsLooking for a needle in a haystack: a comprehensive study of hallucinations in neural machine translation\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,A\. Vlachos and I\. Augenstein \(Eds\.\),Dubrovnik, Croatia,pp\. 1059–1075\.External Links:[Link](https://aclanthology.org/2023.eacl-main.75/),[Document](https://dx.doi.org/10.18653/v1/2023.eacl-main.75)Cited by:[§1](https://arxiv.org/html/2610.11389#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.11389#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2610.11389#S3.SS1.p1.1)\.
- Gururanganet al\.\(2018\)S\. Gururangan, S\. Swayamdipta, O\. Levy, R\. Schwartz, S\. R\. Bowman, and N\. A\. SmithAnnotation artifacts in natural language inference data\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),New Orleans, Louisiana,pp\. 107–112\.External Links:[Link](https://aclanthology.org/N18-2017/),[Document](https://dx.doi.org/10.18653/v1/N18-2017)Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p3.1)\.
- Heet al\.\(2023\)P\. He, J\. Gao, and W\. ChenDeBERTaV3: improving DeBERTa using ELECTRA\-style pre\-training with gradient\-disentangled embedding sharing\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=sE7-XhLxHA)Cited by:[§3\.5](https://arxiv.org/html/2610.11389#S3.SS5.p1.1),[§4\.1](https://arxiv.org/html/2610.11389#S4.SS1.p1.1)\.
- Heffernanet al\.\(2022\)K\. Heffernan, O\. Çelebi, and H\. SchwenkBitext mining using distilled sentence representations for low\-resource languages\.InFindings of the Association for Computational Linguistics: EMNLP 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 2101–2112\.External Links:[Link](https://aclanthology.org/2022.findings-emnlp.154/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-emnlp.154)Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p2.1)\.
- Honnibal and Montani \(2017\)M\. Honnibal and I\. MontanispaCy: industrial\-strength natural language processing in Python\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.1212303)Cited by:[§3\.2\.2](https://arxiv.org/html/2610.11389#S3.SS2.SSS2.p1.1)\.
- Jalili Sabetet al\.\(2020\)M\. Jalili Sabet, P\. Dufter, F\. Yvon, and H\. SchützeSimAlign: high quality word alignments without parallel training data using static and contextualized embeddings\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 1627–1643\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.147/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.147)Cited by:[§2\.1](https://arxiv.org/html/2610.11389#S2.SS1.p1.1)\.
- Jiet al\.\(2023\)Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, A\. Madotto, and P\. FungSurvey of hallucination in natural language generation\.ACM Computing Surveys55\(12\),pp\. 1–38\.External Links:[Document](https://dx.doi.org/10.1145/3571730)Cited by:[§1](https://arxiv.org/html/2610.11389#S1.p1.1)\.
- Kimet al\.\(2017\)H\. Kim, J\. Lee, and S\. NaPredictor\-estimator using multilevel task learning with stack propagation for neural quality estimation\.InProceedings of the Second Conference on Machine Translation,O\. Bojar, C\. Buck, R\. Chatterjee, C\. Federmann, Y\. Graham, B\. Haddow, M\. Huck, A\. J\. Yepes, P\. Koehn, and J\. Kreutzer \(Eds\.\),Copenhagen, Denmark,pp\. 562–568\.External Links:[Link](https://aclanthology.org/W17-4763/),[Document](https://dx.doi.org/10.18653/v1/W17-4763)Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p1.1)\.
- Leeet al\.\(2018\)K\. Lee, O\. Firat, A\. Agarwal, C\. Fannjiang, and D\. SussilloHallucinations in neural machine translation\.InNeurIPS 2018 Workshop on Interpretability and Robustness in Audio, Speech, and Language,Montréal, Canada\.External Links:[Link](https://openreview.net/pdf?id=SkxJ-309FQ)Cited by:[§1](https://arxiv.org/html/2610.11389#S1.p1.1)\.
- Lewiset al\.\(2020\)M\. Lewis, Y\. Liu, N\. Goyal, M\. Ghazvininejad, A\. Mohamed, O\. Levy, V\. Stoyanov, and L\. ZettlemoyerBART: denoising sequence\-to\-sequence pre\-training for natural language generation, translation, and comprehension\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 7871–7880\.External Links:[Link](https://aclanthology.org/2020.acl-main.703/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.703)Cited by:[§2\.1](https://arxiv.org/html/2610.11389#S2.SS1.p1.1)\.
- Liuet al\.\(2020\)Y\. Liu, J\. Gu, N\. Goyal, X\. Li, S\. Edunov, M\. Ghazvininejad, M\. Lewis, and L\. ZettlemoyerMultilingual denoising pre\-training for neural machine translation\.Transactions of the Association for Computational Linguistics8,pp\. 726–742\.External Links:[Link](https://aclanthology.org/2020.tacl-1.47/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00343)Cited by:[§3\.7](https://arxiv.org/html/2610.11389#S3.SS7.SSS0.Px4.p1.1)\.
- Martindaleet al\.\(2019\)M\. Martindale, M\. Carpuat, K\. Duh, and P\. McNameeIdentifying fluently inadequate output in neural and statistical machine translation\.InProceedings of Machine Translation Summit XVII: Research Track,M\. Forcada, A\. Way, B\. Haddow, and R\. Sennrich \(Eds\.\),Dublin, Ireland,pp\. 233–243\.External Links:[Link](https://aclanthology.org/W19-6623/)Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p1.1)\.
- NLLB Teamet al\.\(2022\)NLLB Team, M\. R\. Costa\-jussà, J\. Cross, O\. Çelebi,et al\.No language left behind: scaling human\-centered machine translation\.arXiv preprint arXiv:2207\.04672\.External Links:[Link](https://arxiv.org/abs/2207.04672)Cited by:[§3\.2\.2](https://arxiv.org/html/2610.11389#S3.SS2.SSS2.Px5.p1.1),[§3\.7](https://arxiv.org/html/2610.11389#S3.SS7.SSS0.Px4.p1.1),[§4\.4](https://arxiv.org/html/2610.11389#S4.SS4.SSS0.Px1.p1.1)\.
- Papineniet al\.\(2002\)K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,P\. Isabelle, E\. Charniak, and D\. Lin \(Eds\.\),Philadelphia, Pennsylvania, USA,pp\. 311–318\.External Links:[Link](https://aclanthology.org/P02-1040/),[Document](https://dx.doi.org/10.3115/1073083.1073135)Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p1.1)\.
- Poliaket al\.\(2018\)A\. Poliak, J\. Naradowsky, A\. Haldar, R\. Rudinger, and B\. Van DurmeHypothesis only baselines in natural language inference\.InProceedings of the Seventh Joint Conference on Lexical and Computational Semantics,New Orleans, Louisiana,pp\. 180–191\.External Links:[Link](https://aclanthology.org/S18-2023/),[Document](https://dx.doi.org/10.18653/v1/S18-2023)Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p3.1)\.
- Ranathungaet al\.\(2024\)S\. Ranathunga, N\. De Silva, V\. Menan, A\. Fernando, and C\. RathnayakeQuality does matter: a detailed look at the quality and utility of web\-mined parallel corpora\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Y\. Graham and M\. Purver \(Eds\.\),St\. Julian’s, Malta,pp\. 860–880\.External Links:[Link](https://aclanthology.org/2024.eacl-long.52)Cited by:[§3\.2\.1](https://arxiv.org/html/2610.11389#S3.SS2.SSS1.p1.1)\.
- Raunaket al\.\(2021\)V\. Raunak, A\. Menezes, and M\. Junczys\-DowmuntThe curious case of hallucinations in neural machine translation\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 1172–1183\.External Links:[Link](https://aclanthology.org/2021.naacl-main.92/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.92)Cited by:[§2\.2](https://arxiv.org/html/2610.11389#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2610.11389#S3.SS1.p1.1)\.
- Reiet al\.\(2020\)R\. Rei, C\. Stewart, A\. C\. Farinha, and A\. LavieCOMET: a neural framework for MT evaluation\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 2685–2702\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.213/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.213)Cited by:[§2\.3](https://arxiv.org/html/2610.11389#S2.SS3.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§1](https://arxiv.org/html/2610.11389#S1.p1.1)\.
- Wang and Sennrich \(2020\)C\. Wang and R\. SennrichOn exposure bias, hallucination and domain shift in neural machine translation\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 3544–3552\.External Links:[Link](https://aclanthology.org/2020.acl-main.326/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.326)Cited by:[§1](https://arxiv.org/html/2610.11389#S1.p2.1)\.
- Xueet al\.\(2021\)L\. Xue, N\. Constant, A\. Roberts, M\. Kale, R\. Al\-Rfou, A\. Siddhant, A\. Barua, and C\. RaffelMT5: a massively multilingual pre\-trained text\-to\-text transformer\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 483–498\.External Links:[Link](https://aclanthology.org/2021.naacl-main.41/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.41)Cited by:[§3\.7](https://arxiv.org/html/2610.11389#S3.SS7.SSS0.Px4.p1.1)\.
- Zhanget al\.\(2020\)T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. ArtziBERTScore: evaluating text generation with BERT\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkeHuCVFDr)Cited by:[§3\.2\.2](https://arxiv.org/html/2610.11389#S3.SS2.SSS2.Px5.p1.1)\.
- Zhouet al\.\(2019\)C\. Zhou, X\. Ma, D\. Wang, and G\. NeubigDensity matching for bilingual word embedding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 1588–1598\.External Links:[Link](https://aclanthology.org/N19-1161/),[Document](https://dx.doi.org/10.18653/v1/N19-1161)Cited by:[§2\.1](https://arxiv.org/html/2610.11389#S2.SS1.p1.1)\.
- Zhouet al\.\(2021\)C\. Zhou, G\. Neubig, J\. Gu, M\. Diab, F\. Guzmán, L\. Zettlemoyer, and M\. GhazvininejadDetecting hallucinated content in conditional neural sequence generation\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 1393–1404\.External Links:[Link](https://aclanthology.org/2021.findings-acl.120/),[Document](https://dx.doi.org/10.18653/v1/2021.findings-acl.120)Cited by:[§2\.1](https://arxiv.org/html/2610.11389#S2.SS1.p1.1)\.