Do LLMs Make More Mistakes If They Do Not Believe the Input Data?

# Do LLMs Make More Mistakes If They Do Not Believe the Input Data?
Source: [https://arxiv.org/html/2609.09363](https://arxiv.org/html/2609.09363)
Aleš Manuel PapáčekVojtěch DvořákOndřej DušekAffiliation:Charles University, Faculty of Mathematics and PhysicsAffiliation:Institute of Formal and Applied LinguisticsAffiliation:Prague, Czech RepublicEmail:[\{kochelka,papacek,dvorak,odusek\}@ufal\.mff\.cuni\.cz](mailto:)

###### Abstract

Large language models \(LLMs\) are prone to hallucinating or misinterpreting facts, which impairs their usability in retrieval\-augmented generation or data\-to\-text systems\. We analyse how faithfulness of LLMs to provided context depends on how plausible they perceive the context to be \(context–memory conflict\)\. To better identify error patterns, we make use of the increased difficulty of non\-English and low\-resource language text generation and input data based on local knowledge, only partially captured in models’ parametric knowledge\. We let the models generate text in English, Czech, Slovak and Upper Sorbian from factual \(FA\), counterfactual \(CFA\) and fictional \(FI\) RDF triples containing local Czech and Slovak data\. Contrary to our expectations, we observe only a weak context–memory conflict on the human\-annotated sample\. For Kimi K3 as an LLM judge, which agrees well with human annotations on the sample, counterfactual inputs receive only slightly lower faithfulness scores than factual ones \(−0\.05\-0\.05on a 1–5 scale\)\. We also find that a suboptimal choice of LLM judge would lead to overestimating the strength of the context–memory conflict\.

## 1Introduction

LLMs can generate fluent text from structured inputs with little task\-specific training, as shown by[Axelsson and Skantze \(2023\)](https://arxiv.org/html/2609.09363#bib.bib9)for GPT\-3\.5 and improving with newer, larger models\. However, fluent output can still omit, alter or add facts from the provided data[Kasner and Dusek \(2024\)](https://arxiv.org/html/2609.09363#bib.bib1)\.

[Xu et al\. \(2024\)](https://arxiv.org/html/2609.09363#bib.bib6)describe the*context–memory conflict*, where a model’s parametric knowledge conflicts with the provided context\. A model may correct or ignore information it perceives as erroneous\. To analyse this tension, we focus on counterfactual and fictional RDF111Triples in the form ’subject \| predicate \| object’inputs\.

Existing benchmarks may not fully reveal this tension due to training data leakage\([Song et al\., 2025](https://arxiv.org/html/2609.09363#bib.bib4)\)\. We therefore create factual \(FA\), counterfactual \(CFA\) and fictional \(FI\) groups of RDF triples from local \(Czech and Slovak\) knowledge data\([Libovický et al\., 2026](https://arxiv.org/html/2609.09363#bib.bib2)\), as local knowledge is less likely to be captured by models’ parametric knowledge\. We ask LLMs to classify each input as FA, FI or CFA\. This way, we obtain input plausibility as perceived by the model, which allows us to compare its behaviour based on the context\-memory conflict strength\. We then generate sentences from these triple groups in English, Czech, Slovak and Upper Sorbian using open\-weight LLMs of various sizes, and we analyse differences based on perceived \(and actual\) input factuality\.

We find that larger models classify input factuality more accurately\. With Kimi K3 as the primary LLM judge based on its agreement with human annotations, we observe only a mild context–memory conflict \(rating counterfactuals items only0\.050\.05points lower on a 1–5 scale compared to factual\)\. However, DeepSeek V4 Pro Preview reports the same effect three to four times larger, so a wrong choice of LLM judge can lead to overestimating the strength of the context–memory conflict\. We release our code and data including annotations at[https://github\.com/pkochelka/counterfactual\-faithfulness](https://github.com/pkochelka/counterfactual-faithfulness)\.

## 2Related Work

[Axelsson and Skantze \(2023\)](https://arxiv.org/html/2609.09363#bib.bib9)showed that prompted LLMs can verbalize knowledge graphs; this also expands to under\-resourced languages[Lorandi and Belz \(2024\)](https://arxiv.org/html/2609.09363#bib.bib3)\. The GEM 2024 shared task\([Mille et al\., 2024](https://arxiv.org/html/2609.09363#bib.bib10)\)evaluated factual, counterfactual, and fictional inputs\.[Sun et al\. \(2026\)](https://arxiv.org/html/2609.09363#bib.bib5)show that model behaviour with respect to the context–memory conflict\([Xu et al\., 2024](https://arxiv.org/html/2609.09363#bib.bib6)\)depends on the task framing, and that models may still rely on parametric knowledge even when instructed to use the context\. Studying this is difficult with established benchmarks, whose data may appear in pretraining\([Song et al\., 2025](https://arxiv.org/html/2609.09363#bib.bib4)\)\. To counteract this,[Kasner and Dusek \(2024\)](https://arxiv.org/html/2609.09363#bib.bib1)evaluate data\-to\-text generation using data collected on\-the\-fly\.[Libovický et al\. \(2026\)](https://arxiv.org/html/2609.09363#bib.bib2)and[Hupkes and Bogoychev \(2025\)](https://arxiv.org/html/2609.09363#bib.bib8)show a similar effect on question\-answering by using relatively obscure local knowledge of non\-English\-speaking nations\. For evaluation, LLM judges can offer better correlation than older automatic metrics[Xu et al\. \(2023\)](https://arxiv.org/html/2609.09363#bib.bib25)but this varies across tasks[Bavaresco et al\. \(2025\)](https://arxiv.org/html/2609.09363#bib.bib24)and is prone to biases[Zheng et al\. \(2023\)](https://arxiv.org/html/2609.09363#bib.bib7)\.

Our work combines the above perspectives of multilingual and low\-resource data\-to\-text, context\-memory conflicts and less familiar data by measuring whether a model’s own perception of factuality affects its faithfulness when verbalizing factual, counterfactual, and fictional RDF inputs\. We use LLM judges audited on human\-annotated samples\.

## 3Data

We use the Czech and Slovak portions of the local\-knowledge CUS\-QA benchmark\([Libovický et al\., 2026](https://arxiv.org/html/2609.09363#bib.bib2)\)\. It is built from native\-speaker questions grounded in local Wikipedia content\. We convert it into RDF triples and add counterfactual and fictional versions \(see Appendix Figure[1](https://arxiv.org/html/2609.09363#A1.F1)\)\.

#### Converting CUS\-QA into triples

We feed question\-answer pairs to Mistral Medium 3\.5\([Mistral AI, 2026](https://arxiv.org/html/2609.09363#bib.bib19)\), prompting it to generate a statement and corresponding triples for each pair\. Each subject and object in the resulting triples is assigned a type by gpt\-oss\-120b\([OpenAI, 2025](https://arxiv.org/html/2609.09363#bib.bib14)\)using a constrained label set \(e\.g\., personal name, place, date, number\)\. Gpt\-oss\-120b is given context from Wikipedia to improve type accuracy\. The result is our factual \(FA\) dataset\.

#### Counterfactual and fictional sets

are created following[Mille et al\. \(2024\)](https://arxiv.org/html/2609.09363#bib.bib10)\. Counterfactual \(CFA\) data are built from factual triples by randomly substituting subjects and objects with other entities of the same type\. Fictional \(FI\) data are created similarly, using fictional entities generated by Claude Sonnet 5\([Anthropic, 2026a](https://arxiv.org/html/2609.09363#bib.bib23)\)and gpt\-oss\-120b\.

The resulting dataset contains 2,847 instances, each composed of 1\-9 RDF triples: 1,482 with Czech local data \(530 FA, 476 CFA, and 476 FI\) and 1,365 with Slovak \(493 FA, 436 CFA, and 436 FI\)\.

## 4Experiments

#### Classification\.

To test models’ parametric knowledge, we prompt them to classify the input set of triples into FA/CFA/FI \(cf\. Section[3](https://arxiv.org/html/2609.09363#S3)\)\. The classification is repeated five times with a temperature of 1\.0, the majority label is used as the final prediction\. We also measure model consistency \(proportion of unanimous classifications\)\.

#### Data\-to\-text generation\.

We prompt the LLMs to verbalize each RDF triple set as one or a few natural sentences in the target language, using greedy decoding\.

Full prompts for both tasks are in Appendix[F](https://arxiv.org/html/2609.09363#A6)\.

### 4\.1Experimental Settings

We use Czech and Slovak as the data languages and Czech, Slovak, English, and Upper Sorbian as the prompt/target languages\. Czech and Slovak let us compare prompts that match the language/locale of the input data \(“Same local”\) with prompts in the other closely related language \(“Other local”\)\. English represents a very high\-resource language, whereas Upper Sorbian represents a language closely related to Czech and Slovak but very low\-resource\.222Upper Sorbian is estimated to have around 25,000 speakers\([Minority Rights Group International,](https://arxiv.org/html/2609.09363#bib.bib20)\)\.The experiment thus runs along four dimensions: model \(nine models described in Section[4\.2](https://arxiv.org/html/2609.09363#S4.SS2)\), data language \(CS,SK\), example variant \(CFA,FA,FI; Section[3](https://arxiv.org/html/2609.09363#S3)\), and prompt/target language \(CS,SK,EN,HSB\), yielding 11,388 generations per model\.

### 4\.2Tested Models

We evaluate nine open\-weight generator models from five model families, separated into three groups:*Large*\(\>100B\) – Qwen3\.5 122B\-A10B[Qwen Team \(2026\)](https://arxiv.org/html/2609.09363#bib.bib11)and gpt\-oss\-120b[OpenAI \(2025\)](https://arxiv.org/html/2609.09363#bib.bib14);*Medium*\(\>4B\) – Gemma 4 31B and E4B[Gemma Team \(2026\)](https://arxiv.org/html/2609.09363#bib.bib13), Llama 4 Scout 17B\-16E Instruct[Meta AI \(2025\)](https://arxiv.org/html/2609.09363#bib.bib15)and Qwen3\.5 9B;*Small*– Gemma 4 E2B, Tiny Aya Global 3\.35B[Salamanca and others \(2026\)](https://arxiv.org/html/2609.09363#bib.bib16)and Qwen3 1\.7B[Yang et al\. \(2025\)](https://arxiv.org/html/2609.09363#bib.bib12)\. Configuration details are in Appendix[C](https://arxiv.org/html/2609.09363#A3)\.

### 4\.3LLM\-as\-Judge Evaluation

As the full dataset is too large to score manually, we use LLM judges to rate each generated text from 1 to 5 forfaithfulnessto the input RDF triples andfluencyin the target language\.

#### Faithfulness

measures agreement with the source RDF triples, penalizing omissions and unsupported claims\. We also test how closely the model followed the instructions to provide coherent text in the target language, penalizing outputs in a different language \(e\.g\. Czech instead of Slovak\) or non\-natural language \(e\.g\., leaving triples as\-is\)\. The judge also identifies the triple\(s\) that were not verbalized correctly and provides a short comment\.

#### Fluency

measures only how natural\-sounding the sentences are in the target language, regardless of how faithful they are\. The judge is not provided the input triples in order to focus on fluency only; however, flattened entities from the source triples are provided to adjust expectations on proper names\.

Full prompts are in Appendix[F](https://arxiv.org/html/2609.09363#A6)\. We report mean scores and bootstrap 95% confidence intervals\.333For population\-level mean scores, brackets report 95% confidence intervals from 10,000 source\-item cluster\-bootstrap resamples\. Generated sentences from the same source triples are kept together during resampling\. Paired contrasts instead resample matched dataset–item clusters\.

### 4\.4Manual Annotation for Judge Tuning

To obtain reference labels against which we can tune and audit the LLM judge prompt, we pick a small tuning sample \(540 instances\) for manual annotation by the authors\. We balance it by factuality, model and language\. We picked DeepSeek V4 Pro Preview[DeepSeek\-AI \(2026\)](https://arxiv.org/html/2609.09363#bib.bib17)and ran it with an initial prompt to also balance the sample with respect to faithfulness scores \(see Appendix[B](https://arxiv.org/html/2609.09363#A2)\)\. We hid the initial judge scores and LLM identifiers for annotation\. We also annotated a heldout sample of 108 instances for additional LLM judge evaluation, using the same stratified design\. We then iteratively tuned the judge prompt with DeepSeek on the tuning sample to more make it closely follow the guidelines and match the human labels\.

### 4\.5Judge Validation

Using the final optimized prompt on both the tuning and heldout samples, we evaluated DeepSeek as well as four alternatives: Kimi K3[Kimi Team and Moonshot AI \(2026\)](https://arxiv.org/html/2609.09363#bib.bib18), Claude Opus 5[Anthropic \(2026b\)](https://arxiv.org/html/2609.09363#bib.bib21)with a low reasoning effort and GPT\-5\.6 Sol[OpenAI \(2026\)](https://arxiv.org/html/2609.09363#bib.bib22)with low and high reasoning\.

Table 1:Agreement with human faithfulness labels\.κw\\kappa\_\{w\}is quadratic\-weighted kappa, Exact and≤1\\leq 1are percentages, and MAE is mean absolute error\. Parentheses show the reasoning effort setting\.Table[1](https://arxiv.org/html/2609.09363#S4.T1)shows agreement with human annotation of faithfulness\. Claude Opus 5 agrees best on both samples, but is prohibitively expensive at full scale\. Kimi offers the best balance of agreement and cost\. Its agreement remains stable on the heldout sample \(κw\\kappa\_\{w\}0\.712→\\rightarrow0\.711\), whereas DeepSeek degrades substantially \(0\.555→\\rightarrow0\.350\), suggesting overfitting and weak generalizability\. We thus pick Kimi as our primary judge\.

We further compared Kimi and DeepSeek on fluency annotation, with Kimi agreeing better than DeepSeek \(κw\\kappa\_\{w\}0\.553 versus 0\.456\)\. Both fluency judges are stricter than the annotators \(bias−0\.50\-0\.50for Kimi and−0\.72\-0\.72for DeepSeek\), and this is language\-dependent\. Absolute fluency levels and small cross\-language fluency differences should therefore be read with caution\. Appendix[D](https://arxiv.org/html/2609.09363#A4)provides additional details on judge validation\.

## 5Results

### 5\.1Classification

Table[2](https://arxiv.org/html/2609.09363#S5.T2)shows that most models classify the majority of inputs as FA or CFA; Tiny Aya is the exception, classifying most as FI\. Small models show limited knowledge of the local facts, with 33–36% accuracy, close to the∼33%\{\\sim\}33\\%random baseline\. As the model size grows, so does accuracy\.

ModelAcc\.Unan\.FA/FI/CFAFaith\.Flu\.Qwen3\.5 122B52\.081\.640/9/514\.904\.15GPT OSS 120B54\.264\.436/16/484\.963\.98\[0\.5pt/2pt\] Gemma4 31B52\.898\.555/6/384\.943\.76Llama4 17B48\.471\.256/11/334\.774\.09Qwen3\.5 9B47\.328\.453/19/284\.844\.13Gemma4 E4B37\.589\.992/4/44\.463\.49\[0\.5pt/2pt\] Gemma4 E2B36\.293\.494/1/54\.122\.84Tiny Aya 3\.35B33\.119\.39/78/133\.663\.74Qwen3 1\.7B35\.353\.574/20/63\.493\.00Table 2:Classification and generation results\. Acc\. = exact majority\-vote classification accuracy, Unan\. = % of examples with unanimous classification in five runs\. FA/FI/CFA is the percentage distribution of majority predictions across classes \(excl\. 59 unparseable classification attempts\)\. Faith\. and Flu\. = mean Kimi K3 scores on a 1–5 scale over all 11,388 outputs per model\. Model size groups are split by dashed lines\.
### 5\.2Generation

Table[2](https://arxiv.org/html/2609.09363#S5.T2)reports the results of each tested model including faithfulness and fluency as given by the Kimi K3 judge\. Faithfulness generally increases with model size, while fluency follows a less consistent pattern\. Faithfulness also generally decreases with the number of input triples under both judges \(see Appendix Table[15](https://arxiv.org/html/2609.09363#A5.T15)\)\.

Table[3](https://arxiv.org/html/2609.09363#S5.T3)presents Kimi faithfulness by prompt\-language condition \(cf\. Section[4\.1](https://arxiv.org/html/2609.09363#S4.SS1)\)\. English and matching local\-language prompts perform best, non\-matching Czech and Slovak prompts perform slightly worse, and Upper Sorbian is the most difficult, especially for smaller models\. Upper Sorbian accounts for 50\.4% of faithful\-but\-disfluent cases, and large models retain mean faithfulness 4\.85 in Upper Sorbian while fluency falls to 3\.53, a 1\.32\-point gap compared with 0\.54 in English\. Similarly, changing from matching to the other local language drops faithfulness by 0\.03 but fluency by 0\.29\.

Table 3:Mean Kimi K3 faithfulness by prompt\-language condition and model\-size group\. Brackets show 95% confidence intervals\.[3](https://arxiv.org/html/2609.09363#footnote3)
### 5\.3Factual, Fictional and Counterfactual

We now compare the three input variants directly\. Of the 1,023 factual items, 912 also have a fictional and a counterfactual counterpart, so we restrict this analysis to those\. Under Kimi as the judge, the score ordering is FA \> FI \> CFA in all model groups and overall, and 95%*pairwise*confidence intervals from resampling dataset–item clusters 10,000 times exclude zero for every contrast\. The effect is nonetheless tiny \(cf\. Table[4](https://arxiv.org/html/2609.09363#S5.T4)\): the CFA and FI faithfulness penalties are−0\.049\-0\.049and−0\.030\-0\.030on a 1–5 scale\. The same ordering holds for fluency in every model\-size group, and the counterfactual rewrite costs marginally more fluency than faithfulness \(−0\.071\-0\.071versus−0\.049\-0\.049\)\. DeepSeek produces substantially larger CFA–FA differences in both faithfulness \(−0\.177\-0\.177\) and fluency \(−0\.198\-0\.198\), but we regard Kimi as the more reliable judge\.

Table 4:Mean Kimi faithfulness by true data variant and model\-size group on the 32,832 matched FA/FI/CFA outputs, pooling both datasets and all four prompt languages\. Brackets show 95% confidence intervals\.[3](https://arxiv.org/html/2609.09363#footnote3)#### Judge robustness checks\.

The judge prompt does not say if an input is FA, FI or CFA\. Disclosing this raises DeepSeek’s scores by0\.1480\.148, and its agreement with the human labels drops \(κw\\kappa\_\{w\}0\.592→0\.4990\.592\\rightarrow 0\.499\)\. Kimi shifts only by0\.0500\.050\(κw\\kappa\_\{w\}0\.717→0\.7090\.717\\rightarrow 0\.709\), within rerun noise\. Asking either judge explicitly to classify the sentence it is scoring changes little: the judges neither recover the factuality condition reliably, nor penalise sentences they believe counterfactual beyond what the annotators do\. Appendix[D](https://arxiv.org/html/2609.09363#A4)has details on both checks\.

#### Perceived factuality\.

Table[5](https://arxiv.org/html/2609.09363#S5.T5)examines whether faithfulness varies with how generation models themselves perceive the input, using classification vote shares as soft assignments\. With Kimi as the judge, the ordering differs across model groups, with only small models showing FA \> FI \> CFA strongly\. DeepSeek again shows a larger and less trustworthy effect\. Additional details are given in Appendix[E](https://arxiv.org/html/2609.09363#A5)\.

Table 5:Mean faithfulness scores \(1–5\) by model\-size group and the model’s classification of the input RDF triples, soft\-weighted by classification vote share\. Pred\. denotes the predicted class\. Scores are from Kimi K3; brackets show 95% confidence intervals\.[3](https://arxiv.org/html/2609.09363#footnote3)
#### Judge–human gap reflects factuality\.

The LLM judge and humans score the same outputs, so their difference reflects judge behaviour rather than generator quality\. On the 648 human\-annotated rows, the Kimi–human gap shrinks from\+0\.35\+0\.35on FA to\+0\.25\+0\.25on FI to\+0\.16\+0\.16on CFA \(or from\+0\.33\+0\.33to\+0\.28\+0\.28and\+0\.15\+0\.15if grouping by perceived classes\)\.

#### Explicit conflict acknowledgement\.

As the base generation prompt does not explicitly mention conflicts with prior knowledge, we re\-run the generation with an alternative prompt, instructing models to treat all triples as authoritative and avoid correction, hedging, refusal or disclaimers\. The pooled change is only−0\.012\-0\.012on the 1–5 scale, with no effect on ordering \(see Appendix Table[19](https://arxiv.org/html/2609.09363#A5.T19)\)\. 88% of all outputs get the same score and nearly half are even textually identical\. The largest score changes only reflect formatting changes in Upper Sorbian \(e\.g\., inclusion of ‘\|’ separators\)\.

### 5\.4Error analysis

To better understand the faithfulness scores, we group the judge’s error annotations into six non\-exclusive categories: use of incorrect language or copying input triples unchanged \(Lang\), hallucinating unsupported information \(Hall\), reversing relation directions \(Rev\), missing source facts \(Miss\), incorrect relations between correct entities \(Rel\), adding/substituting an incorrect entity \(Ent\)\. Detailed definitions and examples are in Appendix[G](https://arxiv.org/html/2609.09363#A7)\.

Table[6](https://arxiv.org/html/2609.09363#S5.T6)reports Kimi’s structured error labels\. We observe about twice as many hallucinations for FI \(highest\) and CFA compared to FA data in both medium and small\-sized models, whereas the results are generally stable for large models\. No error category follows the FA–FI–CFA ordering consistently across model sizes\.

Table 6:Kimi K3 error\-category incidence \(% outputs\) by predicted class and model size; categories are non\-exclusive\. The number of outputs in each row varies with the model prediction distribution in Table[2](https://arxiv.org/html/2609.09363#S5.T2)\.However, some CFA errors explicitly reveal the conflict between the provided data and the LLMs’ parametric knowledge: despite instructions to use only the input triples and preserve all values exactly, Llama4\-Scout generated the historically accurate year 1620 when the Battle of White Mountain was given the counterfactual date 2013, and rejected 1848 as singer Karel Kryl’s album release date because it preceded his birth\. These examples show that LLMs may favour memorised knowledge over the input, giving plausible but unfaithful outputs\.

## 6Conclusion

Across nine open\-weight models and four languages, larger generators are generally more faithful and all models struggle most in Upper Sorbian\. However, we find little support for the context–memory\-conflict story\. On matched FA\-CFA items, the CFA faithfulness penalty is−0\.049\-0\.049under our primary Kimi K3 judge, about one percent of the scale, and the FA\>\>FI\>\>CFA ordering it preserves is numerical rather than substantive\. The CFA penalty is in fact more pronounced for fluency than faithfulness\. An explicit instruction to treat contradictory triples as authoritative does not reduce the penalty either\. Validated against human labels, the judges are themselves mildly kinder to factual inputs, by several times the residual gap\. We therefore read the headline context–memory\-conflict effect in this setting as small and not cleanly separable from evaluator bias and the general difficulty of realizing atypical inputs\.

The broader lesson concerns evaluation rather than generation\. On exactly the same outputs, DeepSeek V4 Pro Preview puts the counterfactual penalty at−0\.177\-0\.177, three to four times higher, which on its own would have supported a clean context–memory\-conflict claim about the generators\. Nothing in that judge’s aggregate agreement with our annotators would have flagged the problem \(κw\\kappa\_\{w\}0\.555, 80\.9% of scores within one point\): a judge can agree well overall and still carry a bias aligned with the experimental condition, and that bias is then easily read as a property of the generators\. Manual annotation held out from prompt tuning \(κw\\kappa\_\{w\}0\.350\), and evaluating alternative judges, is what separates a property of the models from a property of the measurement\.

## Acknowledgments

This work was funded by the European Union \(ERC, NG\-NLG, 101039303\), the Czech AI Factory \(CZAI\) project \(10131474, EuroHPC JU\), and project CZ\.02\.01\.01/00/23\_020/0008518 of the Czech Ministry of Education\. We also acknowledge support of the Czech Ministry of Education through the e\-INFRA CZ research infrastructure \(MetaCentrum & CERIT\-SC, ID:90254\)\. We thank the anonymous reviews for their helpful comments, especially motivating us to use a better judge\.

## Limitations

The main limitations of our work we are aware of are the following:

- •None of the authors participating in manual annotation for judge tuning was an Upper Sorbian speaker\. However, all were native or near\-native speakers of Czech and Slovak and given the languages’ similarity, they were able to check the factuality of Upper Sorbian texts\.
- •We tuned the LLM\-judge prompt on the 540\-row sample and report agreement on it\. The sample is stratified partly by an earlier judge score and has one author label per row without independent adjudication, so those numbers may overestimate generalization\. The disjoint 108\-row sample, annotated after the prompt was frozen tries to fix this little bit\.
- •To speed up generation, we ran all of the models with reasoning off or with a low reasoning effort, which may alter their behaviour compared to higher reasoning modes\.

## References

- Anthropic \(2026a\)AnthropicClaude Sonnet 5\.Note:[https://www\.anthropic\.com/claude](https://www.anthropic.com/claude)Large language model, accessed August 17, 2026Cited by:[§3](https://arxiv.org/html/2609.09363#S3.SS0.SSS0.Px2.p1.1)\.
- Anthropic \(2026b\)AnthropicIntroducing Claude Opus 5\.Note:[https://www\.anthropic\.com/news/claude\-opus\-5](https://www.anthropic.com/news/claude-opus-5)Accessed: 2026\-08\-29Cited by:[§4\.5](https://arxiv.org/html/2609.09363#S4.SS5.p1.1)\.
- Axelsson and Skantze \(2023\)A\. Axelsson and G\. SkantzeUsing large language models for zero\-shot natural language generation from knowledge graphs\.InProceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge \(MM\-NLG 2023\),A\. Gatt, C\. Gardent, L\. Cripwell, A\. Belz, C\. Borg, A\. Erdem, and E\. Erdem \(Eds\.\),Prague, Czech Republic,pp\. 39–54\.External Links:[Link](https://aclanthology.org/2023.mmnlg-1.5/)Cited by:[§1](https://arxiv.org/html/2609.09363#S1.p1.1),[§2](https://arxiv.org/html/2609.09363#S2.p1.1)\.
- Bavarescoet al\.\(2025\)A\. Bavaresco, R\. Bernardi, L\. Bertolazzi, D\. Elliott, R\. Fernández, A\. Gatt, E\. Ghaleb, M\. Giulianelli, M\. Hanna, A\. Koller, A\. Martins, P\. Mondorf, V\. Neplenbroek, S\. Pezzelle, B\. Plank, D\. Schlangen, A\. Suglia, A\. K\. Surikuchi, E\. Takmaz, and A\. TestoniLLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 238–255\.External Links:[Link](https://aclanthology.org/2025.acl-short.20/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-short.20),ISBN 979\-8\-89176\-252\-7Cited by:[§2](https://arxiv.org/html/2609.09363#S2.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-V4\-Pro model card\.Hugging Face\.External Links:[Link](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro)Cited by:[§4\.4](https://arxiv.org/html/2609.09363#S4.SS4.p1.1)\.
- Gemma Team \(2026\)Gemma TeamGemma 4 technical report\.External Links:2607\.02770,[Document](https://dx.doi.org/10.48550/arXiv.2607.02770),[Link](https://arxiv.org/abs/2607.02770)Cited by:[§4\.2](https://arxiv.org/html/2609.09363#S4.SS2.p1.1)\.
- Hupkes and Bogoychev \(2025\)D\. Hupkes and N\. BogoychevMultiLoKo: a multilingual local knowledge benchmark for llms spanning 31 languages\.External Links:2504\.10356,[Link](https://arxiv.org/abs/2504.10356)Cited by:[§2](https://arxiv.org/html/2609.09363#S2.p1.1)\.
- Kasner and Dusek \(2024\)Z\. Kasner and O\. DusekBeyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data\-to\-Text Generation\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 12045–12072\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.651)Cited by:[§1](https://arxiv.org/html/2609.09363#S1.p1.1),[§2](https://arxiv.org/html/2609.09363#S2.p1.1)\.
- Kimi Team and Moonshot AI \(2026\)Kimi Team and Moonshot AIKimi K3: open frontier intelligence\.arXiv preprint arXiv:2607\.24653\.External Links:[Link](https://arxiv.org/abs/2607.24653)Cited by:[§4\.5](https://arxiv.org/html/2609.09363#S4.SS5.p1.1)\.
- Libovickýet al\.\(2026\)J\. Libovický, J\. Helcl, A\. Manea, and G\. VicoCUS\-QA: Local\-Knowledge\-Oriented Open\-Ended Question Answering Dataset\.arXiv\.External Links:2507\.22752,[Document](https://dx.doi.org/10.48550/arXiv.2507.22752)Cited by:[§1](https://arxiv.org/html/2609.09363#S1.p3.1),[§2](https://arxiv.org/html/2609.09363#S2.p1.1),[§3](https://arxiv.org/html/2609.09363#S3.p1.1)\.
- Lorandi and Belz \(2024\)M\. Lorandi and A\. BelzHigh\-quality Data\-to\-Text Generation for Severely Under\-Resourced Languages with Out\-of\-the\-box Large Language Models\.InFindings of the Association for Computational Linguistics: EACL 2024,Y\. Graham and M\. Purver \(Eds\.\),St\. Julian’s, Malta,pp\. 1451–1461\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-eacl.98)Cited by:[§2](https://arxiv.org/html/2609.09363#S2.p1.1)\.
- Meta AI \(2025\)Meta AILlama 4 Scout 17B\-16E Instruct model card\.Hugging Face\.External Links:[Link](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E)Cited by:[§4\.2](https://arxiv.org/html/2609.09363#S4.SS2.p1.1)\.
- Milleet al\.\(2024\)S\. Mille, J\. Sedoc, Y\. Liu, E\. Clark, A\. J\. Axelsson, M\. A\. Clinciu, Y\. Hou, S\. Mahamood, I\. N\. Obonyo, and L\. ZhangThe 2024 GEM shared task on multilingual data\-to\-text generation and summarization: overview and preliminary results\.InProceedings of the 17th International Natural Language Generation Conference: Generation Challenges,S\. Mille and M\. Clinciu \(Eds\.\),Tokyo, Japan,pp\. 17–38\.External Links:[Link](https://aclanthology.org/2024.inlg-genchal.2/),[Document](https://dx.doi.org/10.18653/v1/2024.inlg-genchal.2)Cited by:[§2](https://arxiv.org/html/2609.09363#S2.p1.1),[§3](https://arxiv.org/html/2609.09363#S3.SS0.SSS0.Px2.p1.1)\.
- \[14\]Minority Rights Group InternationalSorbs in germany\.Note:[https://minorityrights\.org/communities/sorbs/](https://minorityrights.org/communities/sorbs/)Accessed: 2026\-07\-17Cited by:[footnote 2](https://arxiv.org/html/2609.09363#footnote2)\.
- Mistral AI \(2026\)Mistral AIMistral Medium 3\.5 model card\.Mistral AI\.External Links:[Link](https://docs.mistral.ai/models/model-cards/mistral-medium-3-5-26-04)Cited by:[§3](https://arxiv.org/html/2609.09363#S3.SS0.SSS0.Px1.p1.1)\.
- OpenAI \(2025\)OpenAIgpt\-oss\-120b and gpt\-oss\-20b model card\.OpenAI\.External Links:[Link](https://openai.com/index/gpt-oss-model-card/)Cited by:[§3](https://arxiv.org/html/2609.09363#S3.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2609.09363#S4.SS2.p1.1)\.
- OpenAI \(2026\)OpenAIGPT\-5\.6: frontier intelligence that scales with your ambition\.Note:[https://openai\.com/index/gpt\-5\-6/](https://openai.com/index/gpt-5-6/)Accessed: 2026\-08\-29Cited by:[§4\.5](https://arxiv.org/html/2609.09363#S4.SS5.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5 model card\.Hugging Face\.External Links:[Link](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)Cited by:[§4\.2](https://arxiv.org/html/2609.09363#S4.SS2.p1.1)\.
- Salamancaet al\.\(2026\)A\. R\. Salamancaet al\.Tiny Aya: bridging scale and multilingual depth\.External Links:2603\.11510,[Document](https://dx.doi.org/10.48550/arXiv.2603.11510),[Link](https://arxiv.org/abs/2603.11510)Cited by:[§4\.2](https://arxiv.org/html/2609.09363#S4.SS2.p1.1)\.
- Songet al\.\(2025\)Y\. Song, W\. S\. Martinez, A\. Nikiforovskaya, E\. Chapple, and C\. GardentMultilingual Verbalisation of Knowledge Graphs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 1111–1162\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.60),ISBN 979\-8\-89176\-335\-7Cited by:[§1](https://arxiv.org/html/2609.09363#S1.p3.1),[§2](https://arxiv.org/html/2609.09363#S2.p1.1)\.
- Sunet al\.\(2026\)K\. Sun, F\. Bai, and M\. DredzeTask matters: knowledge requirements shape LLM responses to context–memory conflict\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 4154–4176\.External Links:[Link](https://aclanthology.org/2026.findings-acl.202/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.202),ISBN 979\-8\-89176\-395\-1Cited by:[§2](https://arxiv.org/html/2609.09363#S2.p1.1)\.
- Xuet al\.\(2024\)R\. Xu, Z\. Qi, Z\. Guo, C\. Wang, H\. Wang, Y\. Zhang, and W\. XuKnowledge conflicts for LLMs: a survey\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 8541–8565\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.486/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.486)Cited by:[§1](https://arxiv.org/html/2609.09363#S1.p2.1),[§2](https://arxiv.org/html/2609.09363#S2.p1.1)\.
- Xuet al\.\(2023\)W\. Xu, D\. Wang, L\. Pan, Z\. Song, M\. Freitag, W\. Wang, and L\. LiINSTRUCTSCORE: towards explainable text generation evaluation with automatic feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5967–5994\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.365/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.365)Cited by:[§2](https://arxiv.org/html/2609.09363#S2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang,et al\.Qwen3 technical report\.External Links:2505\.09388,[Document](https://dx.doi.org/10.48550/arXiv.2505.09388),[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.2](https://arxiv.org/html/2609.09363#S4.SS2.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.arXiv\.External Links:2306\.05685,[Document](https://dx.doi.org/10.48550/arXiv.2306.05685)Cited by:[§2](https://arxiv.org/html/2609.09363#S2.p1.1)\.

## Appendix AData Creation Pipeline

The schema for the data creation pipeline described in Section[3](https://arxiv.org/html/2609.09363#S3)is shown in Figure[1](https://arxiv.org/html/2609.09363#A1.F1)\.

![Refer to caption](https://arxiv.org/html/2609.09363v1/llm-fa-cf.drawio.png)Figure 1:Pipeline to convert the CUS\-QA dataset into RDF triples\. First, we rewrite question–answer pairs into declarative phrases and decompose them into triples; then assign subject and object types using local Wikipedia context\. We produce the CFA set by randomly swapping entities of the same type, and the FI set by injecting fictional LLM\-generated entities\.
## Appendix BManual Annotation Details

#### Sampling strategy\.

Since the generator models are generally very capable, sampling uniformly would likely result in most examples scoring 5\. We therefore first ran the judging with an initial prompt, to gain some information about which triple groups were challenging in which languages\. Afterwards, we sampled the triple groups and the generated sentences in a way to uniformly cover every variant \(FA, CFA, FI\), every language \(EN, CS, SK, HSB\), three faithfulness score groups \(score 1, scores 2–4, score 5\), and three parameter\-count strata \(below 10B, 10–100B, and above 100B\)\. These strata were used only to construct a diverse annotation sample and differ from the model\-size groups used in the main analysis\. For each combination, we picked 5 random examples, giving us a total of3×4×3×3×5=5403\\times 4\\times 3\\times 3\\times 5=540examples to annotate\. Within the broader faithfulness\-score and model\-size groups, we represented individual scores and models as evenly as possible, maximizing the representation of the least frequent one\. The variant, language and parameter\-count strata are exactly balanced: each of the 36 combinations contributes 15 rows\. The faithfulness\-score strata are not, because outputs scoring 1 are rare for the larger models outside Upper Sorbian, so many cells could not supply 5 of them and were backfilled from the other two strata\. The realized sample therefore contains 66 rows from the score\-1 stratum, 218 from scores 2–4, and 256 from the score\-5 stratum\. The 108\-row held\-out sample was drawn later with the same design and one example per combination, and is likewise balanced on variant, language and parameter count\.

#### Annotation procedure\.

Before annotation, we hid the judge scores and identifiers of the generator models\. The four annotators first agreed on shared annotation guidelines\. Every unclear example encountered afterwards was discussed, and the agreed\-upon decision was used to extend the guidelines\. Finally, we reviewed and updated the scores once more using the completed guidelines, obtaining the author reference labels used to optimize the LLM judge\. Each row nevertheless has one final label rather than independent multi\-annotator adjudication\.

## Appendix CModel Configuration Details

All models used BF16 precision except Qwen3\.5\-122B\-A10B, which used FP8, and gpt\-oss\-120b, which used MXFP4\. We accessed Qwen3\.5\-122B\-A10B, Gemma 4 31B, gpt\-oss\-120b, and Llama 4 Scout through an OpenAI\-compatible cloud service\.444The quantization precision of cloud\-hosted models reflects the deployment configuration available through the respective service\.We ran the remaining models locally via vLLM\. We used the same task\-specific pipelines and prompt templates for all models\.

All models were used with reasoning disabled when the API or backend supported it\. The only exception is gpt\-oss\-120b, where reasoning cannot be fully turned off, so we set its reasoning level to low\.

## Appendix DJudge Robustness and Audit Details

#### DeepSeek language\-condition results\.

Table[7](https://arxiv.org/html/2609.09363#A4.T7)is the direct robustness counterpart to the primary Kimi results in Table[3](https://arxiv.org/html/2609.09363#S5.T3)\. The ordering is identical, although Kimi is higher in every cell except the nearly unchanged small\-model Upper Sorbian result \(1\.43 versus 1\.42\)\.

Table 7:Mean DeepSeek V4 Pro Preview faithfulness by prompt\-language condition and model\-size group on the same outputs as Table[3](https://arxiv.org/html/2609.09363#S5.T3)\.
#### DeepSeek structured error labels\.

Table[8](https://arxiv.org/html/2609.09363#A4.T8)shows that DeepSeek reports an FA\-to\-FI\-to\-CFA rise in hallucination labels for every size group, whereas Kimi yields a different pattern\. Because the judges’ structured labels are less stable than their scores, we interpret the difference as judge sensitivity rather than gold error incidence\.

Table 8:DeepSeek V4 Pro Preview structured error\-category incidence \(% of paired outputs\) by predicted class and model size\. Categories are non\-exclusive\.
#### Direct judge comparison\.

We compared Kimi and DeepSeek directly on both faithfulness and fluency\. Table[9](https://arxiv.org/html/2609.09363#A4.T9)shows differences in faithfulness assessments \(split per language and agreggated over all generating models\), with Kimi generally assigning higher scores\. Table[10](https://arxiv.org/html/2609.09363#A4.T10)then shows a similar breakdown for fluency; here, the agreement between judges is higher overall, with no clear trend in scores\.

Table 9:Direct paired faithfulness comparison\. D is DeepSeek V4 Pro Preview, K is Kimi K3,Δ=K−D\\Delta=\\mathrm\{K\}\-\\mathrm\{D\}, Exact and≤1\\leq 1are the percentages of outputs on which the two judges agree exactly and within one point, andκw\\kappa\_\{w\}is quadratic\-weighted Cohen’s kappa\.Table 10:Paired comparison of DeepSeek V4 Pro Preview \(D\) and Kimi K3 \(K\) fluency scores on all 102,492 outputs\.Δ=K−D\\Delta=\\mathrm\{K\}\-\\mathrm\{D\}, andκw\\kappa\_\{w\}is quadratic\-weighted Cohen’s kappa\. The judges agree exactly on 62\.8% of outputs and within one point on 90\.8%\.
#### Reasoning effort in the audit judges\.

On the same 540 rows, GPT\-5\.6 Sol high and low agree exactly on 87\.8% of scores and within one point on 97\.6%\. High reasoning modestly improves agreement with the human labels while increasing observed reasoning tokens from 35,596 to 75,536\. Agreement figures for all five judges are in Table[1](https://arxiv.org/html/2609.09363#S4.T1)\.

#### Population\-scale maximum disagreements\.

The 3,337 four\-point gaps concentrate on target\-language and raw\-RDF\-format decisions, especially in Upper Sorbian\. Examples include Kimi accepting raw predicate names that DeepSeek treats as an untransformed triple, and Kimi correctly rejecting Czech, English, Polish, or visibly corrupted mixed\-script grammar under another target\-language prompt\. These examples identify rubric boundaries rather than ground\-truth winners; the full sentences, triples, scores, and judge comments are retained in the released disagreement CSV\.

Upper Sorbian is one quarter of the data but contributes 44\.5% of total absolute score difference and 79\.8% of the 3,337 four\-point \(maximum\) gaps\. Of those gaps, 2,246 are DeepSeek 1 versus Kimi 5 and 1,091 run in the other direction\. Figure[2](https://arxiv.org/html/2609.09363#A4.F2)shows the paired scores over the whole grid: 70\.1% of all outputs receive a 5 from both judges, and the disagreement that remains is asymmetric, concentrated in the column where Kimi awards 5\.

![Refer to caption](https://arxiv.org/html/2609.09363v1/judge_score_confusion_matrix.png)Figure 2:Paired faithfulness\-score matrix\. Cells show counts and percentages of all paired outputs; orange outlines mark exact agreement\. Colour intensity uses a power scale so that less frequent disagreements remain visible\.
#### Disclosing the class to the judge\.

The default judge prompt is blind: it never says whether the triples are FA, FI or CFA\. The disclosure run adds one line stating the condition, everything else is identical\. Because a judge rerun is not deterministic at the item level, we also ran the blind prompt a second time under the same configuration\. That*replicate*run is a noise floor, any disclosure effect smaller than it is not measurable\. Table[11](https://arxiv.org/html/2609.09363#A4.T11)reports both\.

Table 11:Effect of disclosing the FA/FI/CFA class of the input to the judge, on the 540\-item annotation sample\.*replicate*is the blind prompt run twice under the same configuration, giving the run\-to\-run noise floor\. Mean shift is against that judge’s blind run, with 10,000\-fold bootstrap percentile intervals over paired per\-item differences;*Changed*is the share of items whose score moves;κw\\kappa\_\{w\}is quadratic\-weighted agreement with the human labels\.DeepSeek’s disclosure effect exceeds its own noise floor on every measure: it moves 23\.5% of items against a floor of 11\.1%, and costs0\.0930\.093of agreement with the humans where a plain rerun costs0\.0240\.024\. The shift is upward, so the disclosed judge is more lenient and*less*accurate at once\. Kimi’s effect does not exceed its noise floor on any measure — its agreement degrades less under disclosure \(−0\.008\-0\.008\) than under a plain rerun \(−0\.022\-0\.022\), and it moves 11\.1% of items against a floor of 8\.9%\. The two lifts also have inverted shapes: DeepSeek’s concentrates on CFA \(\+0\.25\+0\.25, FA flat\), Kimi’s on FA \(\+0\.106\+0\.106, CFA flat at\+0\.006\+0\.006\)\.

#### The class the judge perceives\.

Separately from the scoring runs, and without altering the judge prompt, we asked each judge to label the generated sentences of the annotation sample as FA, FI or CFA, given the same sentence and triples it sees when scoring\. The faithfulness scores below are the judges’ ordinary scores for those items, only the grouping is new\.

Table[12](https://arxiv.org/html/2609.09363#A4.T12)shows that neither judge can reliably predict the factuality, and that they fail in opposite directions: DeepSeek defaults to FA, calling 44\.7% of counterfactual sentences factual, while Kimi defaults to CFA, calling 44\.4% of factual sentences counterfactual\.

Table 12:How each judge classifies the generated sentence when shown the same sentence and triples it scores, row\-normalised over the true condition, 540 items\. DeepSeek is biased towards FA and Kimi towards CFA\. Bold marks each judge’s most frequent label per row\.Table[13](https://arxiv.org/html/2609.09363#A4.T13)groups each judge’s scores by its own perceived class, with the human mean on the same items as an anchor\. DeepSeek’s scores are flat across its perceived classes: items it calls CFA score0\.0950\.095below those it calls FA \(CI includes zero\) and the humans separate the same items by0\.0530\.053\. Kimi does separate its perceived classes \(−0\.328\-0\.328\), but by*less*than the humans do on those same items \(−0\.567\-0\.567\), so the gap reflects a real quality difference that Kimi under\-weights rather than a penalty for perceived counterfactuality\. In neither case does believing a sentence to be counterfactual make a judge harsher than a human on that sentence\.

Table 13:Mean score by the judge’s own perceived class of the sentence, with the human mean over the annotated subset of the same items\.

## Appendix EAdditional Results

This section complements the aggregate results in the main paper with analyses by input size, uncertainty estimates for individual models, and detailed results based on the ground\-truth and perceived classes\.

#### Joint true/perceived results\.

Table[14](https://arxiv.org/html/2609.09363#A5.T14)gives the primary Kimi results behind the main\-text summary\.

Table 14:Mean Kimi K3 faithfulness by ground\-truth and model\-perceived input class\. Scores are macro\-averaged across models and soft\-weighted by the five classification votes; brackets show 95% confidence intervals\.
#### DeepSeek joint true/perceived results\.

For direct comparison with the primary Kimi table, DeepSeek gives 4\.41, 4\.42, and 4\.22 for true FA across perceived FA/FI/CFA; 4\.30, 4\.35, and 4\.14 for true FI; and 4\.26, 4\.30, and 4\.10 for true CFA\. Unlike Kimi, true CFA is lowest within all three perceived columns, although the differences are small\.

#### Faithfulness by number of input triples\.

Table[15](https://arxiv.org/html/2609.09363#A5.T15)gives the per\-model results behind the aggregate trend reported in the main paper\. Scores generally decrease with the number of triples under both judges\. DeepSeek is monotonic for every model; Kimi is nearly flat for GPT OSS 120B and rises from one to 2–3 triples for Gemma4 E2B\.

Table 15:Average faithfulness by number of input triples\. D and K denote DeepSeek V4 Pro Preview and Kimi K3\. Values pool both datasets, all prompt languages, and all three variants\.
#### Confidence intervals for overall results\.

Table[16](https://arxiv.org/html/2609.09363#A5.T16)reports uncertainty estimates for both judges on the complete data\. The intervals are narrow throughout, so the per\-model orderings in Table[2](https://arxiv.org/html/2609.09363#S5.T2)are not sampling noise\. Faithfulness broadly tracks model size under both judges, whereas the fluency ordering is less regular and ranks Llama4 Scout and Qwen3\.5 9B close to or above both large models\.

Table 16:Mean Kimi K3 and DeepSeek V4 Pro Preview faithfulness and fluency scores by generator\. Brackets show 95% confidence intervals\.
#### Per\-model results by perceived class\.

Table[17](https://arxiv.org/html/2609.09363#A5.T17)expands the model\-size results in Table[5](https://arxiv.org/html/2609.09363#S5.T5)into individual models, with confidence intervals in place of the effective sample sizes we previously used to signal which cells are thin\. Under Kimi, five of nine models \(Qwen3\.5 122B, Llama4 Scout, Qwen3\.5 9B, Gemma4 E4B and Qwen3 1\.7B\) have a Pred\. FA advantage over Pred\. CFA with disjoint intervals, marginally so for Qwen3\.5 9B\. The differences are much smaller than under DeepSeek, and two models reverse the direction: Gemma4 E2B scores 4\.09 versus 4\.54 and Tiny Aya 3\.17 versus 3\.45, again with disjoint intervals; GPT OSS 120B and Gemma4 31B overlap\. Under DeepSeek the FA\-over\-CFA direction holds for seven of nine models, with Gemma4 E2B and Tiny Aya again reversed\.

Table 17:Mean faithfulness grouped by the generator’s self\-classification and soft\-weighted by its five classification votes\. Brackets show 95% confidence intervals; J is the judge: D is DeepSeek V4 Pro Preview and K is Kimi K3\.
#### Significance of perceived\-class differences\.

Under DeepSeek, perceived CFA is more difficult in every model\-size group\. For large, medium, and small models, respectively, the Bonferroni\-adjusted 95% family\-wise confidence intervals for Pred\. FA−\-Pred\. CFA were \[0\.09, 0\.15\], \[0\.04, 0\.11\], and \[0\.35, 0\.47\]; the corresponding intervals for Pred\. FI−\-Pred\. CFA were \[0\.01, 0\.10\], \[0\.05, 0\.14\], and \[0\.25, 0\.37\]\. Under Kimi, the medium\-group direction differs and the large\-group differences shrink to near\-ceiling margins\. We therefore interpret the DeepSeek intervals as judge\-specific robustness results rather than the primary conclusion\.

#### A proprietary generator as a reference point\.

The nine evaluated models are all open\-weight\. To place their scores on a scale, we additionally ran the full generation and classification pipeline with DeepSeek V4 Pro Preview as a*generator*and judged its outputs with Kimi K3\. Table[18](https://arxiv.org/html/2609.09363#A5.T18)gives the result\. It is excluded from every other table in this paper because we have only Kimi scores for it: we avoid self\-evaluation, so DeepSeek does not judge its own outputs\.

Two observations follow\. First, its mean faithfulness of 4\.92 is comparable to the best open\-weight models, and it holds that level in Upper Sorbian \(4\.93, slightly above the 4\.85 of the large open\-weight group and far above the 1\.43 of the small one\)\. The gap to the small models isolates target\-language competence rather than faithfulness as the binding constraint at the low end\. Second, its spread across the three data variants is 0\.03 points, smaller than any open\-weight model’s, while its classification accuracy of 49\.3% is below that of gpt\-oss\-120b — so recognizing counterfactual input and being derailed by it are clearly separable abilities\.

ConditionFaithfulnessOverall4\.92 \[4\.91, 4\.93\]VariantFA4\.94 \[4\.92, 4\.95\]FI4\.92 \[4\.90, 4\.93\]CFA4\.91 \[4\.89, 4\.92\]LanguageEN4\.98 \[4\.97, 4\.98\]CS4\.84 \[4\.81, 4\.86\]SK4\.94 \[4\.93, 4\.95\]HSB4\.93 \[4\.91, 4\.94\]Table 18:DeepSeek V4 Pro Preview used as a*generator*over all 11,388 inputs, judged by Kimi K3 only\. Brackets show 95% confidence intervals\. Its majority classification accuracy is 49\.3% with 81\.9% unanimity, predicting FA/FI/CFA on 69/18/13% of inputs\.
#### Conflict\-Acknowledgement Generation Ablation

Table[19](https://arxiv.org/html/2609.09363#A5.T19)shows results comparing the default prompt \(Base\) with an alternative prompt explicitly acknowledging potential conflicts with parametric knowledge \(Ack\.; see Appendix[F\.3](https://arxiv.org/html/2609.09363#A6.SS3)\)\.

Table 19:Conflict\-acknowledgment generation\-prompt ablation judged by Kimi K3\.Δ\\Delta= acknowledgment minus baseline; Sco\. A\. = score agreement; Idnt\. O\. = percentage of identical generations\.
#### Per\-language mistakes analysis\.

Table[20](https://arxiv.org/html/2609.09363#A5.T20)compares the error categories defined in Section[G](https://arxiv.org/html/2609.09363#A7)across prompt\-language conditions under both judges, on the same explicit\-label mapping as Tables[6](https://arxiv.org/html/2609.09363#S5.T6)and[8](https://arxiv.org/html/2609.09363#A4.T8), using the prompt\-language conditions defined in Section[5\.2](https://arxiv.org/html/2609.09363#S5.SS2)\. Each condition has the same number of outputs\.

Table 20:Error\-category incidence \(% of paired outputs\) by prompt\-language condition\. Each cell is DeepSeek / Kimi; categories are non\-exclusive\. Each condition has 25,623 outputs\.The main difference is the high incidence of language or transformation errors with HSB prompts, and it is the one error signal on which the two judges nearly agree: 37\.6% of HSB outputs contain such errors under DeepSeek and 36\.2% under Kimi, against 0\.1% for EN under both\. Small models often failed to produce a sentence in Upper Sorbian, resulting in faithfulness and fluency scores near the bottom of the scale, as shown earlier in Table[3](https://arxiv.org/html/2609.09363#S5.T3)\. The other categories are far less stable: Kimi reports at most a third of DeepSeek’s hallucination rate in every condition, while reporting three to four times as many omissions outside Upper Sorbian\. This is the same judge sensitivity that the main text reports by predicted class\.

## Appendix FExperimental Prompt Templates

This appendix provides the English versions of the prompt templates used in the experiments\. The classification and data\-to\-text generation prompts were used in four prompt languages: English, Czech, Slovak, and Upper Sorbian\. The language\-specific versions differ only in the language of the instructions and output requirements\. Braced expressions denote values filled in for each example, such as the number of triples, the data category, the target language, or the generated sentence\.

### F\.1Classification Prompt

Classify a set of RDF triples as exactly one of these labels:\- FA \(factual\): the information is true and matches real\-world knowledge\.\- CFA \(counterfactual\): the entities are real, but one or more stated relationships or values are false or altered\.\- FI \(fictional\): the entities themselves are fictional or do not exist in the real world\.Use only these three labels\. Classify the whole set with one label, not each triple separately\. You may reason internally if needed, but do not put reasoning in the final answer\.Below are \{size\} RDF triples from the thematic domain ’\{domain\}’\.\{triples\_str\}Output only the label: FA, CFA, or FI\.

### F\.2Data\-to\-Text Generation Prompt

You will be given RDF triples and must express them as fluent English prose\.Each triple is in subject \| predicate \| object order\.Generate a single fluent English sentence \(or two short ones if needed\) that naturally expresses ALL of the given facts\.Strict requirements:\- Use ONLY the information given in the triples\. Do NOT add any facts, details, attributes, or context that are not explicitly stated\.\- Do NOT omit any fact from the triples\.\- Do NOT infer, assume, embellish, or generalize beyond what the triples literally state\.\- Preserve all numbers, dates, and values exactly as given\.\- Translate any entity that is NOT a proper named entity into English \(e\.g\. common nouns, roles, generic descriptions\)\. Keep proper named entities \(names of people, places, organisations, works, etc\.\) in their original form\.\- ALWAYS translate the predicate into English and use it naturally in the sentence\.Output ONLY the sentence\(s\), nothing else\.Below are \{size\} RDF triples from the thematic domain ’\{domain\}’: \{triples\_str\}

### F\.3Conflict\-Acknowledgment Addition

For the ablation in Section[5\.3](https://arxiv.org/html/2609.09363#S5.SS3.SSS0.Px4), we add the following paragraph to the English generation prompt; equivalent Czech, Slovak, and Upper Sorbian versions are used for the other target languages\.

> Important: Some inputs are intentionally designed to contradict common knowledge or your own prior knowledge\. For this task, treat every provided triple as authoritative, even if it appears false, implausible, or inconsistent with what you know\. Express the claims exactly as given; do not correct them, replace them with known facts, hedge, refuse, or add a disclaimer\.

### F\.4Faithfulness\-Judge Prompt

You are judging faithfulness in data\-to\-text generation\. Decide whether the sentence is fully supported by the triples below\. The triples are the complete and ONLY source of truth\. Do not use outside knowledge\. Do not explain your reasoning\. Output the final JSON immediately\.STEP 0 — GATES \(check these before judging content\):\- Wrong language / wrong transform, set faithfulness\_score = 1 with one incorrect\_information entry \(comment label: wrong\_language or wrong\_transform\):\- if the sentence’s own grammar — its verbs, function words, and connecting phrases — is not written mostly in \{target\_language\}\. Judge this only on the words the sentence adds around the triple terms, never on wording copied from the triples\. Be especially careful if target language is low\-resource, such as upper serbian \(hsb\), as even a technically valid sentence in different language should be scored low\. However, if only parts of the sentence are not in the target language, while most of it is, be a bit more benevolent, and follow instructions below\.\- Any subject or object wording taken from the triples — named entities \(people, places, organizations, titles, works\) as well as descriptive or common\-noun terms — routinely remains in its original language inside a correctly written \{target\_language\} sentence\. This alone is NEVER wrong\_language and never justifies score 1, even when such copied wording makes up most of the sentence’s length\. If a translatable term was translated, left untranslated or translated imprecisely, it is at most a fluency problem and never a reason for lowering faithfulness score\.\- if it is not a natural\-language sentence \(e\.g\., it keeps the exact \| pipes from the source triple\)\. If it just keeps the RDF order, or does not split the predicate, it is alright, continue with instructions below\- Degenerate repetition: if the sentence loops or repeats the same clause/phrase at least 10 times, the final score is at most 3 \(lower if it violates additional rules for content faithfulness\); add an entry with comment label: degenerate\. For info\_used on this entry, do NOT quote the repeated span verbatim \-\- write a short placeholder like "\(repeated/looping text\)" instead, then continue directly to faithfulness\_score\. Minor redundancy is acceptable\.CORE RULES:\- Treat every triple as true, even if it is nonsense in the real world\.\- NEVER penalize implausibility of any kind\. All of the following are FULLY FAITHFUL \(score 5\) if they match the triples:\- self\-referential statements \(e\.g\., "X is located in X\."\)\- impossible or reversed time spans \(e\.g\., "from 2000 to 1900"\)\- geographically, historically, or otherwise logically absurd facts\- Interpret each triple as subject \| predicate \| object, using the predicate label’s actual semantics\. Some labels define the object’s role: X \| broadcastedBy \| Y means Y broadcast X; X \| riverFlowingThrough \| Y means river Y flows through X; X \| hasToItsNorth \| Y means Y is north of X\.\- Reversed relation = the sentence swaps the SEMANTIC roles of subject and object, so the resulting claim asserts the opposite of what the triple states\. Do not mark a relation reversed just because the sentence order differs: many of the target languages \(e\.g\. Czech, Slovak\) mark grammatical role through case endings and agreement rather than word position, so a subject can correctly appear after the object\. Determine the role each entity plays from grammar/meaning in context, not from which one appears first, and only flag reverse when the asserted relationship itself is actually flipped\.\- Paraphrase is fine: any wording with the same meaning counts as supported\. In particular, X \| country \| Y is often fully supported by "X is located in Y", "X is in Y", "X lies in Y" and similar\. Apply the same tolerance to other predicates\.\- Name translation is fine: translating descriptive or common\-noun names into \{target\_language\} is acceptable, as are inflection, diacritics, and transliteration changes\. Only penalize a name change if it alters the entity’s identity or meaning\.\- Do not penalize word order, function words, or stylistic differences when the meaning is preserved\.\- Omission: if any information from any triple is missing from the sentence, count it as an omitted claim \(info\_used = "\(missing\)", correct\_info = the omitted triple\)\.\- Only place a claim in incorrect\_information if it is unsupported, contradicted, reversed, negated, omitted, over\-specific, or incorrectly transformed relative to the triples\. NEVER place supported information there\.SCORING RUBRIC \(after gates\):\- 5 = fully faithful: every claim explicitly supported \(paraphrase allowed\), nothing unsupported, nothing omitted — even if nonsensical, self\-referential, or implausible\. If incorrect\_information is \[\], the score MUST be 5\.\- 4 = one minor bug: exactly one wrong predicate or object, one small unsupported claim, one omitted claim, minor translation bug or similar; everything else correct\.\- 3 = mixed: several problems, or supported content mixed with a clear substantive error, or a degenerate/looping sentence \(cap\)\.\- 2 = reversed relation, negated/contradicted predicate, or multiple major problems\.\- 1 = wrong language, kept original pipe\-separated\-RDF format, or the sentence is largely unsupported / contradicts the triples overall\.Return STRICT JSON with these keys only:\- incorrect\_information: array of objects with keys info\_used, correct\_info, comment\- info\_used: the exact claim from the sentence that is wrong/unsupported \("\(missing\)" for omissions\)\- correct\_info: the triple\-backed correction or missing constraint\- comment: brief reason, starting with one label if applicable: unsupported \| wrong\_object \| wrong\_predicate \| reverse \| negation \| omission \| over\_specific \| wrong\_language \| wrong\_transform \| repetition\- faithfulness\_score: integer from 1 to 5Mini examples:1\. Triple: Havlíčkův Brod \| riverFlowingThrough \| Hrubý JeseníkSentence: The Hrubý Jeseník river flows through Havlíčkův Brod\.Output idea: faithfulness\_score 5, incorrect\_information \[\]2\. Target language: EnglishTriples:\- Nejsevernější národní park v České republice \| hasName \| budova hlavního nádraží v Praze\- budova hlavního nádraží v Praze \| isLocatedIn \| Česká republika\- budova hlavního nádraží v Praze \| hasProperty \| nejsevernějšíSentence: The northernmost national park in the Czech Republic is named the main railway station building in Prague\. The main railway station building in Prague is located in the Czech Republic and has the property of being the northernmost\.Output idea: faithfulness\_score 5, incorrect\_information \[\] — every claim matches its triple exactly\. It does not matter that a park being "named" a building is real\-world nonsense; do not reinterpret the subject/object relationship to make it more sensible\.3\. Target language: EnglishTriples:\- Československá automobilová doprava \| providesService \| autobusová doprava\- Československá automobilová doprava \| operatesIn \| ČeskoslovenskoSentence: Československá automobilová doprava provides autobusová doprava in Československo\.Output idea: faithfulness\_score 5, incorrect\_information \[\] — the sentence’s own grammar \("provides \.\.\. in \.\.\."\) is entirely English; the Czech subject/object wording is copied straight from the triples, which is never wrong\_language, no matter how much of the sentence’s length it takes\.4\. Target language: EnglishTriple: Kateřina Bobková\-Valentová \| isBrotherOf \| Svatý VáclavSentence: \.\.\.Kateřina Bobková\-Valentová is his brother\.\.\.Output idea: faithfulness\_score 5, incorrect\_information \[\] — "X isBrotherOf Y" means X is the brother of Y, so "Kateřina Bobková\-Valentová is his \(Svatý Václav’s\) brother" states exactly that relationship\. Do not flip the roles based on which name a pronoun like "his" seems to attach to in the surrounding context — check only whether the sentence’s claim matches the triple’s own subject/object direction\. Only call this reverse if the sentence instead said Svatý Václav is Kateřina’s brother\.5\. Target language: Upper Sorbian \(hsb\)Triple: Zlatá kopa \| originatesIn \| Národný park Nízke TatrySentence: Zlatá kopa \| originatesIn \| Národný park Nízke TatryOutput idea: faithfulness\_score 1, comment "wrong\_transform" — the sentence is literally the raw triple, not a natural\-language sentence in any language\.6\. Target language: EnglishTriples:\- okres Beroun \| namedAfter \| těžba stříbrných rud\- okres Beroun \| namedAfter \| tavení stříbraSentence: The okres Beroun is named after both the mining of silver ores and the smelting of silver\.Output idea: faithfulness\_score 5, incorrect\_information \[\] — "the mining of silver ores" and "the smelting of silver" are faithful English paraphrases of the triples’ Czech object phrases, not unsupported additions\. Do NOT require the sentence to reuse the triple’s exact source wording — judge the meaning, not the surface form, and do not flag a term as unsupported merely because it was translated or phrased differently than the triple\.7\. Target language: EnglishTriples:\- Najvýchodnejší slovenský národný park \| name \| Národný park Poloniny\- Národný park Poloniny \| location \| Čierny Balog\- Národný park Poloniny \| position \| najvýchodnejšíSentence: The most eastern Slovak national park, named Národný park Poloniny, is located in Čierny Balog and is the most eastern\.Output idea: faithfulness\_score 5, incorrect\_information \[\] — attaching a "name" triple to its subject via apposition \("The most eastern Slovak national park, named X, \.\.\."\) and then continuing with the subject’s other facts is a normal way to combine several triples about the same entity into one sentence\. This is NOT merging the subject and object into a single entity or an unsupported claim — every fact still traces to its own triple\.8\. Target language: SlovakTriples:\- První parostrojní železnice v českých zemích \| hasName \| Severní dráha císaře Ferdinanda\- Severní dráha císaře Ferdinanda \| isLocatedIn \| Nové SadySentence: Severní dráha císaře Ferdinanda je názov prvej parostrojnej železnice v českých zemiach, ktorá sa nachádza v Nových Sadoch\.Output idea: faithfulness\_score 5, incorrect\_information \[\] — the sentence’s own grammar \("je názov", "ktorá sa nachádza v"\) is Slovak; "Severní dráha císaře Ferdinanda" and "První parostrojní železnice v českých zemích" are entity names copied straight from the triples, in their original Czech form\. Copied triple wording never counts toward "is the sentence mostly in \{target\_language\}" — not even when, as here, it is most of the sentence’s visible text\. This is NOT wrong\_language\.Target language: \{target\_language\}Category: \{category\}Sentence: \{sentence\}Triples:\{modified\_triples\}

### F\.5Fluency\-Judge Prompt

You are judging the linguistic quality of a single sentence written in a given target language\.Task: decide quickly how well the sentence is written in that language\.Judge only the language: grammar, inflection and agreement, word order, spelling and orthography, punctuation, word choice, and how natural it sounds to a native speaker\.Do not explain your reasoning\.Output the final JSON immediately\.Return STRICT JSON with these keys only:\- fluency\_score: integer from 1 to 5\- fluency\_comment: a brief explanation of the language quality, in EnglishImportant rules:\- Ignore whether the sentence is true, plausible, counterfactual, or nonsensical\. Content correctness is NOT your concern — only linguistic quality\.\- A sentence can be perfectly fluent even if what it says is false or impossible\.\- Example: "Praha je nejvyšší hora na Slovensku\." is completely fluent Czech and must score 5, even though it is factually false\.\- Do not reward or penalise a sentence for the amount of information it contains\.\- Judge the sentence as \{language\}\. If it is written in a different language, that is a fluency failure\.\- Some entities may not be in \{language\}: proper named entities \(names of people, places, organisations, works, etc\.\) may remain in \{entity\_language\} rather than being translated\. Do NOT treat such untranslated proper names as a fluency error — judge only the quality of the surrounding \{language\} text\.\- The source lexical terms below are deduplicated subjects and objects from the source triples, with RDF underscores already replaced by spaces\. Proper names may be preserved as written\. Non\-proper common nouns, adjectives, roles, and descriptions should normally be translated or naturally rendered in \{language\}; awkward untranslated common terms may lower fluency\.Anchors:\- 5 means flawless, natural, native\-quality \{language\}: no grammatical, spelling, or word\-choice issues\- 4 means good: fully understandable with at most a minor slip that a native speaker would barely notice\- 3 means acceptable: understandable, but with noticeable grammatical, spelling, or word\-choice errors\- 2 means poor: frequent or serious errors that make the sentence awkward or hard to read\- 1 means broken: severely ungrammatical, garbled, or not really \{language\}Language: \{language\}Source lexical terms:\{lexical\_terms\}Sentence: \{sentence\}

## Appendix GMistake Categories and Examples

#### Category mapping\.

The mistake categories are derived from the structuredincorrect\_informationentries returned by the faithfulness judges\. The same mapping is applied to both judges’ labels; the examples given below are taken from the DeepSeek V4 Pro Preview annotations\. We consider only outputs with a faithfulness score below 5 when assigning an error type\. The reported percentages in Table[6](https://arxiv.org/html/2609.09363#S5.T6), however, use all judged outputs in the corresponding condition as the denominator\. Thus, they describe the incidence of each error type among all generations, rather than its proportion among erroneous generations only\.

The judge may identify several incorrect claims in one output\. We map the fine\-grained labels attached to these claims into six broader categories:wrong\_language,wrong\_transform, and degenerate repetition are grouped as language or transformation errors \(Lang\);unsupported,hallucination, andover\_specificas hallucination \(Hall\);reverseas a reversed relation \(Rev\);omissionas missing information \(Miss\);wrong\_predicateas an incorrect relation \(Rel\); andwrong\_objectas an incorrect entity or value \(Ent\)\. The categories are non\-exclusive because one output may contain several different mistakes\. Rare contradiction annotations and entries without a recognized label are not included in these six categories\.

#### Examples\.

The following examples illustrate the distinction between the categories\. They are selected from large, medium, and small models rather than from a single model\-size group\.

Lang: language or transformation error\.This category includes outputs written in a language other than the requested one, outputs that retain RDF predicates instead of forming a natural\-language sentence, and strongly degenerate repetitions\. For example,Qwen3\.5\-122Bwas asked to produce an English sentence from:

> Triples:kraj Vysočina∣\\midisA∣\\midhistorická země; kraj Vysočina∣\\midlocation∣\\midseverovýchod České republiky Output:Kraj Vysočina je historická země nacházející se na severovýchodě České republiky\.

The content follows the triples, but the complete sentence is in Czech rather than English\. This is therefore a language error rather than a hallucinated fact\.

Hall: hallucination\.This category covers information not supported by the source triples\. It also includes cases where the model replaces a counterfactual fact with information that is more consistent with its general knowledge\. For example,Tiny\-Ayareceived the counterfactual triple:

> Triple:Rakúsko∣\\midshortName∣\\midPIENAP Output:Rakousko, oficiálně známé jako Rakouská republika, je země v srdci Evropy\. \[Austria, officially known as the Republic of Austria, is a country in the heart of Europe\.\]

Instead of stating thatPIENAPis the short name of Austria, the model ignores the counterfactual relation and produces a factual, general description of Austria\. DeepSeek therefore assigns this output a faithfulness score of 1\.

Rev: reversed relation\.A reversed relation preserves the relevant entities and often the general relation, but exchanges their semantic roles\. For example,Gemma4\-E2Bproduced:

> Triple:Rákosníček∣\\midvoicedBy∣\\midFerdinand d’Este Output:Ferdinand d’Este voiced by Rákosníček\.

The triple states that Rákosníček is voiced by Ferdinand d’Este, while the output makes Rákosníček the voice of Ferdinand d’Este\.

Miss: missing information\.An omission occurs when an entity, relation, or complete source triple is not verbalized\. For example,Qwen3\.5\-122Breceived:

> Triple:Národní divadlo v Praze∣\\midfoundationStoneLaidDate∣\\mid1868\-01\-01 Output:Kámen základní byl položen dne 1868\-01\-01\. \[The foundation stone was laid on 1868\-01\-01\.\]

The date and event are expressed, but the output does not say that they refer to the National Theatre in Prague\. The source entity is therefore missing\.

Rel: incorrect relation\.Here, the output mentions the relevant entities or values but connects them using a different predicate\. For example,Gemma4\-31Breceived:

> Triple:Ľubomír Lisý∣\\midhasField∣\\midneurológia Output:Ľubomír Lisý je neurológ\. \[Ľubomír Lisý is a neurologist\.\]

Having neurology as one’s field does not necessarily state that the person is a neurologist\. The output changes a field\-of\-work relation into an occupation or identity relation\.

Ent: incorrect entity or value\.This category is used when an entity or object from the source is replaced, misidentified, or assigned an incorrect value\. For example,Tiny\-Ayareceived:

> Triple:Slovensko∣\\midnumberOfMinistries∣\\mid15 Output:Slovenia has 15 ministries\.

The number and relation are retained, butSlovensko\(Slovakia\) is replaced with Slovenia, producing a claim about a different country\.

These categories describe the main form of disagreement with the source triples\. They should not be interpreted as fully independent phenomena\. For example, replacing an entity may also make the resulting claim unsupported, and omitting one argument can lead the remaining relation to be interpreted incorrectly\.