When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation

arXiv cs.CL Papers
machine-translation entity-level-mt low-resource evaluation-metrics knowledge-base historical-documents arxiv

Summary

This paper measures the self-referential evaluation loop when a knowledge base is used as the gold standard for entity-level machine translation in low-resource historical domains, showing that gains from KB injection are confined to overlapping segments and do not reflect true translation quality.

arXiv:2608.11843v1 Announce Type: new Abstract: The Seungjeongwon Ilgi, a UNESCO Memory of the World record, is only 37.4% translated, and the most conspicuous failure mode in automatic translation is the person name -- a misread name corrupts the historical fact rather than merely the surface. Low-resource historical domains have no expert gold standard for entity translation, so practitioners substitute a knowledge base (KB) for the gold. That KB is the same resource injected into the system: scoring becomes self-referential and the metric measures instruction compliance rather than translation quality. We measure this loop. Using expert person-name annotations from the National Institute of Korean History as a gold independent of the injection pipeline, we hold the entity set fixed and vary only the provenance of the correct reading. Of 527 expert-annotated mentions, only 31.1% lie outside the injection pipeline, and the residual loop is not uniform -- in the overlapping segment the injected reading agrees with the human translation 97.8% of the time against 70.1% in the independent one, so the segment that looks healthiest is the one the loop is holding up. Across four models, a difference-in-differences analysis shows the gain from KB injection is confined to the segment whose gold shares the injected resource; in the independent segment it is at or below zero. Post-injection preservation clusters in a narrow 0.910-0.996 band even though baseline capability differs fivefold, so the reported gain is the complement of prior performance and weaker models appear to improve more dramatically. On an independent sample built by removing the construction filter, the measure replicates within model (overlapping intervals) while discriminating between models (non-overlapping intervals) -- it reflects a property of the model, not of the sample.
Original Article
View Cached Full Text

Cached at: 08/13/26, 03:29 PM 

# When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation
Source: [https://arxiv.org/abs/2608.11843](https://arxiv.org/abs/2608.11843)
[View PDF](https://arxiv.org/pdf/2608.11843)

> Abstract:The Seungjeongwon Ilgi, a UNESCO Memory of the World record, is only 37\.4% translated, and the most conspicuous failure mode in automatic translation is the person name \-\- a misread name corrupts the historical fact rather than merely the surface\. Low\-resource historical domains have no expert gold standard for entity translation, so practitioners substitute a knowledge base \(KB\) for the gold\. That KB is the same resource injected into the system: scoring becomes self\-referential and the metric measures instruction compliance rather than translation quality\. We measure this loop\. Using expert person\-name annotations from the National Institute of Korean History as a gold independent of the injection pipeline, we hold the entity set fixed and vary only the provenance of the correct reading\. Of 527 expert\-annotated mentions, only 31\.1% lie outside the injection pipeline, and the residual loop is not uniform \-\- in the overlapping segment the injected reading agrees with the human translation 97\.8% of the time against 70\.1% in the independent one, so the segment that looks healthiest is the one the loop is holding up\. Across four models, a difference\-in\-differences analysis shows the gain from KB injection is confined to the segment whose gold shares the injected resource; in the independent segment it is at or below zero\. Post\-injection preservation clusters in a narrow 0\.910\-0\.996 band even though baseline capability differs fivefold, so the reported gain is the complement of prior performance and weaker models appear to improve more dramatically\. On an independent sample built by removing the construction filter, the measure replicates within model \(overlapping intervals\) while discriminating between models \(non\-overlapping intervals\) \-\- it reflects a property of the model, not of the sample\.

## Submission history

From: Jinhyung Bae \[[view email](https://arxiv.org/show-email/43b36a51/2608.11843)\] **\[v1\]**Wed, 12 Aug 2026 09:33:59 UTC \(437 KB\)

Similar Articles

Incentivizing Parametric Knowledge via Reinforcement Learning with Verifiable Rewards for Cross-Cultural Entity Translation

arXiv cs.CL

Researchers from Tianjin University and Alibaba Group propose EA-RLVR, a reinforcement learning framework with verifiable rewards that improves cross-cultural entity translation in LLMs by activating parametric knowledge already encoded during pre-training, without relying on external knowledge bases. Training on 7k samples boosts Qwen3-14B's entity translation accuracy from 23.66% to 31.87% on unseen entities.

Position: Evaluation Scores Are Perishable Knowledge Claims

arXiv cs.AI

This position paper argues that language model evaluation scores should be treated as perishable knowledge claims, not ground truth, and proposes explicit metadata such as formality tier, scope declaration, and expiration date to counter 'trust inflation' caused by averaging weak and strong signals.