Ontology-Driven Structural Regularization for Document-Level Relation Extraction

# Ontology-Driven Structural Regularization for Document-Level Relation Extraction
Source: [https://arxiv.org/html/2608.20856](https://arxiv.org/html/2608.20856)
Laura MenottiAffiliation:Department of Information Engineering, University of Padua, Italy\{name\.surname\}@unipd\.itStefano MarchesinAffiliation:Department of Information Engineering, University of Padua, Italy\{name\.surname\}@unipd\.itGianmaria SilvelloAffiliation:Department of Information Engineering, University of Padua, Italy\{name\.surname\}@unipd\.it

###### Abstract

DocRE\(DocRE\) relies heavily on costly manually annotated datasets, while large distant supervision resources such as*DocRED distant*remain underexploited due to noise\. We show that a critical yet overlooked source of noise lies in structural inconsistencies within relational triples, including violations of ontology constraints and logical contradictions\.

We introduce an ontology\-driven framework to quantify and enforce structural consistency in DocRE datasets\. Our analysis reveals substantial structural noise in DocRED distant and demonstrates that such inconsistencies propagate to model predictions\. Enforcing structural well\-formedness during training significantly reduces logical contradictions and consistently improves generalization performance\. These findings establish structural consistency as a missing axis of supervision in DocRE and highlight structural regularization as an effective strategy for leveraging distant data at scale\.

## 1Introduction

DocRE\(DocRE\) aims to identify all semantic relations holding between entity pairs within a full document\. By extracting relational triples, DocRE enables the construction ofKG, which provide structured, machine\-readable knowledge to support downstream tasks such as information retrieval, question answering, data mining, and recommendation systems[4](https://arxiv.org/html/2608.20856#bib.bib17)\. Building accurate KGs therefore depends on reliable DocRE models trained on high\-quality data\.

The reference benchmark for general\-domain DocRE is DocRED[22](https://arxiv.org/html/2608.20856#bib.bib2), constructed from Wikipedia articles\. DocRED includes both manually annotated datasets and a large distantly supervised dataset\. The manually curated split follows a recommend\-revise paradigm and contains 5,053 documents, divided into 3,053 for training and 1,000 each for development and testing\. In contrast, the DocRED distant dataset comprises 101,873 documents, annotated viaDS\(DS\) by aligning Wikidata triples with Wikipedia text and leveraging BERT\-based named entity annotations\.

While distant supervision enables large\-scale data construction[15](https://arxiv.org/html/2608.20856#bib.bib8), it relies on the strong assumption that if aKGasserts a relation between two entities, then every document mentioning the pair expresses that relation\. This assumption adds substantial noise\. In DocRE, the issue is worse because entities co\-occurring in a document are less likely to express a true relation than those in the same sentence\. As a result, most state\-of\-the\-art models rely mainly on manually annotated data or use distant data only for pre\-training or distillation[19](https://arxiv.org/html/2608.20856#bib.bib5)\.

Recent analyses have further revealed annotation issues even in the manually curated splits\. In particular, around 65% of positive relations were missing in the original DocRED annotations[5](https://arxiv.org/html/2608.20856#bib.bib9), leading to the release of ReDocRED, a revised version of the training and development datasets[20](https://arxiv.org/html/2608.20856#bib.bib10)\. Although existing denoising approaches attempt to mitigate DS noise through uncertainty estimation[18](https://arxiv.org/html/2608.20856#bib.bib6)or active learning for long\-tail relations[14](https://arxiv.org/html/2608.20856#bib.bib12), they focus primarily on semantic correctness and model\-driven pseudo\-label refinement\.

In contrast to model\-centric denoising strategies, we adopt a*data\-centric*approach, focusing on improving the structural quality of training data itself\. We argue that beyond semantic noise, DocRE datasets suffer from*structural inconsistencies*in the extracted triples, such as violations of ontology constraints, missing inverse relations, asymmetric contradictions, and cardinality conflicts\.111We use the term structural consistency to denote the well\-formedness of RDF triples under ontology\-imposed constraints, rather than full ontology satisfiability\.These inconsistencies represent a previously overlooked source of supervision noise that might propagate during model training\.

Concretely, we operationalize structural consistency by leveraging the formal semantics of theOWL\(OWL\) to define explicit structural constraints over entity types and relations\. Our methodology evaluates the*well\-formedness*of relational triples under ontology axioms and quantifies structural violations in both DocRED and ReDocRED, with particular emphasis on the DocRED distant dataset\.

We show that structural noise is substantially more prevalent in distantly supervised data than in manually curated datasets\. In particular, compared to the ReDocRED manual training split, the DocRED distant dataset contains three times more invalid triples, nearly five times more missing inverse relations, and twice as many asymmetric inconsistencies\. These findings suggest that structural inconsistencies constitute a systematic source of noise that can propagate during training, impairing a model’s ability to learn and generate well\-formed triples\. To mitigate this effect, we introduce a lightweight, model\-agnostic*pre\-processing pipeline*that enforces structural consistency prior to training\. The pipeline cleans and augments the dataset without altering model architectures, loss functions, or inference procedures, ensuring seamless integration with existing DocRE systems – whether sequence\-based or graph\-based\.

We show that enforcing ontological consistency acts as a form of structural regularization, improving both prediction coherence and generalization\. Our main contributions are: \(1\) We propose an ontology\-driven framework that usesOWLconstraints to rigorously diagnose ontology\-level structural consistency in DocRE datasets; \(2\) We deliver the first in\-depth quantitative study of structural inconsistencies in DocRED distant and its denoised variants, uncovering extensive, previously unreported violations; \(3\) We show that enforcing structural well\-formedness as a pre\-processing step markedly reduces logical contradictions in model predictions and consistently boosts generalization performance across state\-of\-the\-art DocRE models[25](https://arxiv.org/html/2608.20856#bib.bib4);[11](https://arxiv.org/html/2608.20856#bib.bib3)\.

We release the complete analysis and cleaning framework as open\-source\.222[https://github\.com/mntlra/DocRE\-StructuralConsistency](https://github.com/mntlra/DocRE-StructuralConsistency)The toolkit is compatible with any dataset formatted in the DocRED schema and includes the ontology\-based rules we developed, while allowing for the definition of other custom constraints\.

The rest of this paper is organized as follows\. Section[2](https://arxiv.org/html/2608.20856#S2)reviews prior work on distant denoising and constrainedRE\(RE\)\. Section[3](https://arxiv.org/html/2608.20856#S3)defines the structural rules used in our analysis\. Section[4](https://arxiv.org/html/2608.20856#S4)assesses the structural consistency of DocRED and ReDocRED and examines the main causes of invalid triples\. Section[5](https://arxiv.org/html/2608.20856#S5)studies how structural noise affectsDocREmodels during training\. Finally, Section[6](https://arxiv.org/html/2608.20856#S6)concludes the paper\.

## 2Related Work

DS\(DS\) enables large\-scale relation extraction by aligning entity pairs in text with triples from a referenceKG[15](https://arxiv.org/html/2608.20856#bib.bib8)\. While efficient, this paradigm introduces noisy supervision due to the heuristic alignment between text andKGfacts\.

To address this issue, several works have proposed denoising strategies forDocRE\.[21](https://arxiv.org/html/2608.20856#bib.bib7)introduced a multi\-task framework incorporating mention\-entity matching and fact alignment\.[18](https://arxiv.org/html/2608.20856#bib.bib6)proposed uncertainty\-guided label denoising using Monte Carlo dropout to filter low\-confidence pseudo\-labels\.[14](https://arxiv.org/html/2608.20856#bib.bib12)developed an active learning pipeline tailored to improve long\-tail relation predictions\.

These approaches focus primarily on improving the semantic reliability of distant labels through model\-driven refinement\. In contrast, our work targets a complementary and largely unexplored dimension: structural consistency of relational triples under ontology\-imposed constraints\.

Logical rules have been explored inDocREto enhance relation prediction\.[17](https://arxiv.org/html/2608.20856#bib.bib21)introduced a probabilistic framework that learns logic rules as latent variables and optimizes them viaEM\(EM\)\.[23](https://arxiv.org/html/2608.20856#bib.bib11)proposed a Secondary Reasoning Framework that leverages first\-stage predictions as contextual signals to infer additional relations among neighboring entities\.[24](https://arxiv.org/html/2608.20856#bib.bib22)developed GREP, which models interdependencies between entity pairs to capture fine\-grained reasoning patterns\. These approaches incorporate logical or reasoning mechanisms within the prediction process, focusing on improving semantic inference\. In contrast, our work addresses a complementary dimension: the structural well\-formedness of triples under ontology\-imposed constraints, independently of the model architecture or inference strategy\.

Constraint\-based approaches have also been explored in sentence\-levelREto improve performance under distant supervision\.[9](https://arxiv.org/html/2608.20856#bib.bib18)leveraged entity type information to introduce fine\-grained type constraints for relation prediction\.[6](https://arxiv.org/html/2608.20856#bib.bib20)incorporated Freebase entity descriptions as background knowledge to guide attention mechanisms and improve extraction accuracy\.[8](https://arxiv.org/html/2608.20856#bib.bib19)proposed a*constraint graph*constructed from entity type constraints, which provides an additional supervision signal within a constraint\-aware attention module\. While these methods integrate external knowledge to guide sentence\-level inference, they have not been extended toDocRE, nor do they explicitly enforce ontology\-level structural consistency\.

## 3Methodology

Given a documentddand a set of entitiesE=\{ei\}i=1nE=\\\{e\_\{i\}\\\}\_\{i=1\}^\{n\}, each entityei∈Ee\_\{i\}\\in Eis associated with a set of mentionsMi=\{mi,j\}j=1kiM\_\{i\}=\\\{m\_\{i,j\}\\\}\_\{j=1\}^\{k\_\{i\}\}, wherekik\_\{i\}denotes the number of occurrences ofeie\_\{i\}indd\. The task ofDocREis to predict a subset of relations fromℛ∪\{na\}\\mathcal\{R\}\\cup\\\{\\texttt\{na\}\\\}between entity pairs\(es,eo\)\(e\_\{s\},e\_\{o\}\)fors,o∈\{1,…,n\}s,o\\in\\\{1,\\dots,n\\\}ands≠os\\neq o\. Here,ℛ\\mathcal\{R\}is a pre\-defined set of semantic relations,narepresents the absence of a relation, andes,eoe\_\{s\},e\_\{o\}denote the subject and object entities\.333We adopt the DocRED convention of*subject*and*object*to denote the first and second arguments of an RDF triple ⟨s,p,o⟩, not syntactic roles\.Rather than treating these predictions as independent classification decisions, we interpretDocREas the construction of aKGfocusing on local well\-formedness conditions induced byOWLaxioms\. This perspective allows us to detect structural inconsistencies that are invisible to standard training objectives\.

### 3\.1Invalid Triples

Each entity mentionmi,jm\_\{i,j\}of an entityeie\_\{i\}is classified into a specific type from the set of named entity types𝒯=\{PER, ORG, LOC, TIME, NUM, MISC\}\\mathcal\{T\}=\\\{\\text\{PER, ORG, LOC, TIME, NUM, MISC\}\\\}\. For instance, the mention “*France*” is classified asLOC∈𝒯\\text\{LOC\}\\in\\mathcal\{T\}\. Since all mentions in a set, sayMiM\_\{i\}, identify the same entityeie\_\{i\}, we can assume consistent typing across the set; thus, we use the first mention as representative, i\.e\.,t⁡\(ei\)=t⁡\(mi,1\)t\(e\_\{i\}\)=t\(m\_\{i,1\}\)\.

In this context, the entity pair\(es,eo\)\(e\_\{s\},e\_\{o\}\)and their predicted relationr∈ℛr\\in\\mathcal\{R\}form aRDF\(RDF\) triple⟨s,p,o⟩\\langle s,p,o\\rangle, wheres=ess=e\_\{s\}is the subject,p=rp=ris the predicate, ando=eoo=e\_\{o\}is the object\. Beyond simple triple extraction, we can leverageOWLto define formal semantics over the set of entitiesEEand relationsℛ\\mathcal\{R\}\. Each relationr∈ℛr\\in\\mathcal\{R\}can be viewed as anowl:ObjectPropertywith specific domain and range constraints\. Specifically, for a triple⟨es,r,eo⟩\\langle e\_\{s\},r,e\_\{o\}\\rangleto be semantically valid, the entity types must satisfy:t⁡\(es\)∈dom​\(r\)andt⁡\(eo\)∈ran​\(r\)t\(e\_\{s\}\)\\in\\text\{dom\}\(r\)\\quad\\text\{and\}\\quad t\(e\_\{o\}\)\\in\\text\{ran\}\(r\), wheredom​\(r\),ran​\(r\)⊆𝒯\\text\{dom\}\(r\),\\text\{ran\}\(r\)\\subseteq\\mathcal\{T\}are the permitted subject and object types for relationrr\.

Triples that violate these constraints are consideredinvalid tripleswithin the defined ontology\. By treatingℛ\\mathcal\{R\}as a set of restricted properties rather than arbitrary labels, we can filter out noise and improve the logical consistency of the extracted document graph\.

Each relation in DocRED is mapped to a wikidata property\.444For instance, metadata for property P26 \(*spouse*\) is available at:[https://www\.wikidata\.org/wiki/Property:P26](https://www.wikidata.org/wiki/Property:P26)\.Thus, we leverage the Wikidata property descriptions to identify the domain \(metadata field “subject type constraint”\) and the range of each property \(“value type constraint”\), if present\. We also use the ReDocRED training dataset[20](https://arxiv.org/html/2608.20856#bib.bib10)to extract the most popular entity types of the subject and object entities of each relation and include them in the domain and range\. To avoid overly restrictive criteria, the domain and range of all relations include entity type ”MISC”\. For instance, relation P19 \(*place of death*\) has domain “PER\|\|MISC” and range “LOC\|\|MISC”\.

###### Definition 3\.1\(Invalid Triple\)\.

Letes,eo∈Ee\_\{s\},e\_\{o\}\\in Ebe a pair of entities with associated typest⁡\(es\),t⁡\(eo\)∈𝒯t\(e\_\{s\}\),t\(e\_\{o\}\)\\in\\mathcal\{T\}andr∈ℛr\\in\\mathcal\{R\}be a relation;rris defined asinvalidfor the pair\(es,eo\)\(e\_\{s\},e\_\{o\}\)if the subject entity type or the object entity type is not included in the domain or range of the property:t⁡\(es\)∉dom​\(r\)∨t⁡\(eo\)∉ran​\(r\)t\(e\_\{s\}\)\\notin\\text\{dom\}\(r\)\\quad\\lor\\quad t\(e\_\{o\}\)\\notin\\text\{ran\}\(r\)\.

A triple⟨es,r,eo⟩\\langle e\_\{s\},r,e\_\{o\}\\ranglewhere relationrris invalid for the pair\(es,eo\)\(e\_\{s\},e\_\{o\}\)is called aninvalid triple\. For instance,⟨Emily \(PER\),P19,1987 \(TIME\)⟩\\langle\\text\{Emily \(PER\)\},\\text\{P19\},\\text\{1987 \(TIME\)\}\\rangleis an invalid triple because the object entity type \(TIME\) is not allowed for relation P19 \(*place of death*\), asTIME∉ran​\(P19\)\\text\{TIME\}\\notin\\text\{ran\}\(\\text\{P19\}\)\. The list of domain and range of each relation is reported in Tables[8](https://arxiv.org/html/2608.20856#A3.T8)and[9](https://arxiv.org/html/2608.20856#A3.T9)of Appendix[C](https://arxiv.org/html/2608.20856#A3)\.

### 3\.2Missing Inverse Relations

Inverse properties inOWLallow for the formal definition of bidirectional relationships between entity instances\. Within each document\-level extracted graph, we adopt a local closed\-world assumption with respect to inverse relations: if a relationrrholds between two entities in the document, its declared inverse should also be present in the extracted graph\. We identify a subset of relationsℛi​n​v⊂ℛ\\mathcal\{R\}\_\{inv\}\\subset\\mathcal\{R\}that possess an inverse property\. Formally, for a relationr∈ℛr\\in\\mathcal\{R\}, there may exist a relationr′∈ℛr^\{\\prime\}\\in\\mathcal\{R\}such thatr′≡owl:inverseOfrr^\{\\prime\}\\equiv\\texttt\{owl:inverseOf\}\\,r\. For instance, the relationP​1376P1376\(capital of\) is the inverse ofP​36P36\(capital\)\. To identify these pairs, we leverage the “inverse property” metadata field from the Wikidata properties description\.

###### Definition 3\.2\(Missing Inverse Relation\)\.

Letr,r′∈ℛr,r^\{\\prime\}\\in\\mathcal\{R\}be two relations such thatr′r^\{\\prime\}is declared as theowl:inverseOfrrin the ontology\. By the definition ofowl:inverseOf, the presence of a triple⟨es,r,eo⟩\\langle e\_\{s\},r,e\_\{o\}\\ranglein a documentddentails the existence of the inverse triple⟨eo,r′,es⟩\\langle e\_\{o\},r^\{\\prime\},e\_\{s\}\\rangle\. Hence, a pair of entities\(es,eo\)\(e\_\{s\},e\_\{o\}\)is said to have amissing inverse relationif:

⟨es,r,eo⟩∈d∧⟨eo,r′,es⟩∉d\\langle e\_\{s\},r,e\_\{o\}\\rangle\\in d\\quad\\land\\quad\\langle e\_\{o\},r^\{\\prime\},e\_\{s\}\\rangle\\notin d

Note thatrrandr′r^\{\\prime\}are not required to be distinct; ifr=r′r=r^\{\\prime\}, the relation is defined as anowl:SymmetricProperty, where the predicate must hold in both directions\. The list of inverse relations is reported in Table[10](https://arxiv.org/html/2608.20856#A3.T10)of Appendix[C](https://arxiv.org/html/2608.20856#A3)\.

### 3\.3Asymmetric Relation Inconsistencies

A relationr∈ℛr\\in\\mathcal\{R\}is considered asymmetric if the existence of a relationship in one direction precludes its existence in the opposite direction\. Formally, leveraging the “inverse property” metadata from Wikidata, we identify that the DocRED dataset contains only two symmetric relations:P​26P26\(spouse\) andP​3373P3373\(sibling\), both of which satisfyr≡owl:SymmetricPropertyr\\equiv\\texttt\{owl:SymmetricProperty\}\. All other relationsr∈ℛ∖\{P​26,P​3373\}r\\in\\mathcal\{R\}\\setminus\\\{P26,P3373\\\}are treated as asymmetric\.

###### Definition 3\.3\(Asymmetric Relation Violation\)\.

Letr∈ℛr\\in\\mathcal\{R\}be anowl:AsymmetricProperty\. A triple⟨es,r,eo⟩\\langle e\_\{s\},r,e\_\{o\}\\rangleis defined as anasymmetric triple\. A documentddcontains anasymmetric relation violationif both the triple and its direct inverse \(using the same relationrr\) are present:⟨es,r,eo⟩∈d∧⟨eo,r,es⟩∈d\\langle e\_\{s\},r,e\_\{o\}\\rangle\\in d\\quad\\land\\quad\\langle e\_\{o\},r,e\_\{s\}\\rangle\\in d, wheres≠os\\neq o\. Such a state represents a logical contradiction within theOWLdefinition of asymmetry, as the property cannot be its own inverse unless it is symmetric\.

### 3\.4Relations Cardinality Violations

Given an entity subjectese\_\{s\}, the maximum cardinality of a relationrris defined as the largest possible number of distinct entity objectseoe\_\{o\}such that triples of the form⟨es,r,eo⟩\\langle e\_\{s\},r,e\_\{o\}\\ranglecan be instantiated\. While domain knowledge initially suggests that a subset of1111relations should be strictly functional \(kr=1k\_\{r\}=1\), empirical analysis of the ReDocRED training set reveals that7%7\\%of these relation instances exhibit a maximum cardinality of 2\. This is often due to ground truth comprising both full dates \(e\.g\., “2 July 1931”\) and separate year mentions \(e\.g\., “1931”\) as distinct entities\. For example,15%15\\%of training instances ofP​570P570\(date of death\) presentkr\>1k\_\{r\}\>1\. Consequently, to avoid overly restrictive criteria, we relax the maximum cardinality of such relations tokr=2k\_\{r\}=2\. The complete mapping of relations to their respective maximum cardinalities is provided in Table[11](https://arxiv.org/html/2608.20856#A3.T11)of Appendix[C](https://arxiv.org/html/2608.20856#A3)\.

###### Definition 3\.4\(Cardinality Violation\)\.

Given a relationr∈ℛr\\in\\mathcal\{R\}with maximum cardinalitykrk\_\{r\}, letO⁡\(es,r\)O\(e\_\{s\},r\)be the set of object entities associated with a subjectese\_\{s\}and relationrrin the documentdd:O\(es,r\)=\{eo∈E∣⟨es,r,eo⟩∈d,s≠o\}O\(e\_\{s\},r\)=\\\{e\_\{o\}\\in E\\mid\\langle e\_\{s\},r,e\_\{o\}\\rangle\\in d,s\\neq o\\\}\. Acardinality violationoccurs for the subjectese\_\{s\}if the number of distinct extracted triples exceeds the defined threshold:\|O⁡\(es,r\)\|\>kr\|O\(e\_\{s\},r\)\|\>k\_\{r\}\.

InOWLterms, ifkr=1k\_\{r\}=1, the relationrris treated as anowl:FunctionalProperty; forkr\>1k\_\{r\}\>1, it is treated as aqualified cardinality restriction\.

## 4Structural Consistency in DocRE

We analyze the structural consistency of the DocRED datasets using the rules defined above\. Given DocRED is the reference test collection for the task, we primarily employ the DocRED distant dataset\. We perform the same experiments in the distant datasets produced by the two leading denoising strategies, namely UGDRE[18](https://arxiv.org/html/2608.20856#bib.bib6)and DOREMI[14](https://arxiv.org/html/2608.20856#bib.bib12)\. The results on denoised distant datasets are reported in Appendix[A](https://arxiv.org/html/2608.20856#A1)\.

We analyzed several DocRE datasets from different domains \(especially biomedical\), but none of them was suitable for the study\. BioRED[10](https://arxiv.org/html/2608.20856#bib.bib23)and BC5CDR[7](https://arxiv.org/html/2608.20856#bib.bib24)are two well\-known biomedical DocRE datasets, but they only include one relation type \(“associated\_with"\) between different entity types\. While GutBrainIE[13](https://arxiv.org/html/2608.20856#bib.bib25)includes a sufficient variety of entity types and relations, its preprocessing to remove structural inconsistencies before release renders our analysis infeasible\.

To compare the amount of noise in distantly and manually annotated data, we also consider the DocRED and ReDocRED manual training datasets\. In addition, we perform the same analysis on the ReDocRED evaluation datasets \(dev and test\), which are the revised version of the split DocRED development dataset\. For each dataset, we report the number of errors for each structural constraint \(Section[4\.1](https://arxiv.org/html/2608.20856#S4.SS1)\)\. The code to perform the quality assessment on DocRED and ReDocRED datasets is available online\. We also perform a statistically grounded qualitative analysis on invalid triples in the DocRED distant dataset to understand the main cause of such errors \(Section[4\.2](https://arxiv.org/html/2608.20856#S4.SS2)\)\.

Table 1:Structural consistency analysis of DocRED datasets\. Splits “Train”, “Dev”, and “Test” consider manually annotated datasets, while split “Train DS” reports the quality of the DocRED distant dataset\. For each rule, we report the absolute number of errors in the dataset and its percentage\.### 4\.1Datasets Consistency Assessment

Table[1](https://arxiv.org/html/2608.20856#S4.T1)reports the structural consistency of the DocRED datasets\. If distant supervision introduces systematic structural degradation, we expect higher violations in the DocRED distant split compared to manual datasets\. For each structural constraint defined in Section[3](https://arxiv.org/html/2608.20856#S3), we report both the absolute number of violations and their percentage, computed according to the specific constraint type\.

The percentage of invalid triples \(δ\\delta\) is defined as the ratio between the number of structurally invalid triples and the total number of triples in the dataset\. The percentage of inverse triples missing in a dataset \(σ\\sigma\) is the ratio between the number of missing inverse triples and the triples involving relations with inverses\. The percentage of asymmetric inconsistencies in a dataset \(γ\\gamma\) is the ratio between such inconsistencies and the triples involving an asymmetric relation\. Finally, the percentage of cardinality violations \(θ\\theta\) is the ratio between the subject entity\-relation pair\(es,r\)\(e\_\{s\},r\)violating\|O⁡\(es,r\)\|\>kr\|O\(e\_\{s\},r\)\|\>k\_\{r\}over all such pairs involving a relation with bounded maximum cardinality\.

As expected, DocRED distant presents much more inconsistencies than the manual datasets\. The only exception is on the missing inverse relations, where the worst is the DocRED manual training dataset\. This is a known issue in the literature and has been corrected in ReDocRED[5](https://arxiv.org/html/2608.20856#bib.bib9);[20](https://arxiv.org/html/2608.20856#bib.bib10), as demonstrated by the reduction in missing inverse between the DocRED and ReDocRED manual training dataset \(from around64%64\\%to less than8%8\\%\)\. Compared to the ReDocRED manual training dataset, the DocRED distant dataset contains three times more invalid triples, nearly five times more missing inverse relations, and twice as many asymmetric inconsistencies\. About cardinality violations, DocRED distant is the dataset containing less errors \(0\.01%0\.01\\%\), while the other datasets range between0\.28%0\.28\\%and0\.65%0\.65\\%of violations\. This phenomenon can be attributed to the distant supervision methodology employed in constructing the DocRED distant dataset, which aligns Wikipedia documents with Wikidata triples and thereby preserves their cardinalities\.

Also ReDocRED datasets exhibit structural violations, with a small but non\-negligible proportion of errors in all splits\. Invalid triples constitute between0\.71%0\.71\\%and0\.94%0\.94\\%of the datasets, missing inverse relations remain below10%10\\%, and asymmetric violations range from0\.33%0\.33\\%to0\.48%0\.48\\%\. Cardinality violations exceed those observed in the DocRED manual training and DocRED distant, reaching0\.65%0\.65\\%in the ReDocRED test set\. Given the presence of structural violations in the evaluation datasets, models trained to generate well\-formed triples may exhibit degraded performance\.

### 4\.2Invalid Triples Analysis

To identify the underlying causes of invalid triples in the distant dataset, we conduct a statistically grounded error analysis\. We classify each invalid triple into one of two mutually exclusive categories: \(i\) relation annotation errors, where, given a correctly typed entity pair, the wrong relation type is assigned, and \(ii\) entity annotation errors, where one or both entities are incorrectly typed, thereby inducing an invalid relation\. For example, the triple⟨France​\(𝙻𝙾𝙲\),𝙿𝟷𝟼⁡\(capital\),Paris​\(𝙿𝙴𝚁\)⟩\\mathtt\{\\langle\\textit\{France\}\(LOC\),P16\(\\textit\{capital\}\),\\textit\{Paris\}\(PER\)\\rangle\}is considered an entity annotation error because “*Paris*” is mistakenly classified as a person\.

Since annotating all invalid triples within DocRED distant would be infeasible in terms of time and cost, we instead rely on binomial proportion estimation techniques[2](https://arxiv.org/html/2608.20856#bib.bib1)\. This approach enables us to obtain statistically reliable estimates while limiting the amount of manual annotation required\. The evaluation protocol uses unbiased sampling and estimation procedures, providing strong guarantees on the robustness of the reported results\.

Specifically, we draw a sample usingSRS\(SRS\) of sizen𝒮=400n\_\{\\mathcal\{S\}\}=400from the DocRED distant dataset population of invalid triples, ensuring representativeness of the sample[3](https://arxiv.org/html/2608.20856#bib.bib14)\. Each sampled triple is manually annotated according to the error classification described above\. We formalize the annotation process as a binomial indicator function𝟙​\(⟨es,r,eo⟩\)\\mathds\{1\}\(\\langle e\_\{s\},r,e\_\{o\}\\rangle\), which takes value11if the triple contains an entity annotation error \(e\.g\., incorrect entity type\) and00if it contains a relation annotation error\.

After annotation, we compute the sample proportionμ^=∑⟨es,r,eo⟩∈𝒮𝟙​\(⟨es,r,eo⟩\)n𝒮\\hat\{\\mu\}=\\frac\{\\sum\_\{\\langle e\_\{s\},r,e\_\{o\}\\rangle\\in\\mathcal\{S\}\}\\mathds\{1\}\(\\langle e\_\{s\},r,e\_\{o\}\\rangle\)\}\{n\_\{\\mathcal\{S\}\}\}, which is an unbiased estimator of the true population proportion[3](https://arxiv.org/html/2608.20856#bib.bib14)\. To quantify sampling uncertainty, we accompany this estimate with a1−α1\-\\alphacredible interval\. We employ Highest Posterior Density \(HPD\) intervals[1](https://arxiv.org/html/2608.20856#bib.bib16), which are well suited for uncertainty quantification inKGevaluation settings[12](https://arxiv.org/html/2608.20856#bib.bib13)\. Unlike \(frequentist\) confidence intervals, HPD intervals operate directly in the probabilistic space and avoid common interpretational pitfalls[16](https://arxiv.org/html/2608.20856#bib.bib15), making them appropriate for one\-shot evaluation scenarios[12](https://arxiv.org/html/2608.20856#bib.bib13)\. We setα=0\.05\\alpha=0\.05, yielding a95%95\\%credible interval\.

The analysis shows that69%69\\%\(±4\.5%\\pm 4\.5\\%\) of invalid triples stem from errors in entity annotation, while the remaining31%31\\%are attributable to incorrect relation annotations\. This pronounced disparity demonstrates that entity\-related mistakes are the dominant source of invalid triples\. As a result, the findings imply that enhancing named entity recognition and employing targeted denoising methods for entity annotations could markedly improve the structural reliability of distant supervision datasets\. More broadly, these findings emphasize that entity extraction is a bottleneck for relation extraction\. In end\-to\-end pipelines, inaccuracies in entity recognition propagate to relation prediction, thereby imposing an upper bound on the overall performance ofREsystems\. Consequently, improvements in relation extraction are inherently capped by the quality of entity annotations\. In this work, we do not modify or refine entity extraction; our intervention is limited to enforcing structural consistency over the extracted triples\. As a result, the performance gains we observe should be interpreted as conservative estimates: further improvements could be achieved by jointly addressing entity\-level errors\. Moreover, since entity misannotations are also present in the evaluation data, correcting structurally inconsistent relations may correspond to semantically correct predictions in real\-world settings, yet still be penalized under the current benchmark due to pre\-existing entity annotation errors\.

## 5Training Effect

We investigate if trainingDocREmodels with structurally consistent data improves their performance\.

### 5\.1Cleaning the datasets

We construct a structurally cleaned version of the DocRED distant dataset by eliminating triples that violate the ontological constraints defined in Section[3](https://arxiv.org/html/2608.20856#S3)\. Since cardinality violations are extremely rare across all splits, especially in DocRED distant \(0\.01%0\.01\\%\), we focus our cleaning procedure on invalid triples, missing inverse relations, and asymmetric violations\.

Correcting invalid triples would require manual inspection and revision of both entity types and relation labels\. Given the large number of invalid triples in the DocRED distant dataset \(46,21146\{,\}211\), fully correcting them would entail extensive manual effort and substantial annotation cost\. Instead, we adopt a conservative and scalable strategy: for each invalid triple, we remove the entity involved in the violation from the training set, thereby eliminating all triples associated with that entity within the document\. Under this strategy, removing an entity effectively removes the offending triple\(s\) that depend on its incorrect typing or relation assignment\. This approach is significantly cheaper than manual correction, as it avoids entity re\-labeling while ensuring that no structurally invalid triples remain in the cleaned dataset\. Overall, this procedure results in the removal of208,665208\{,\}665triples and33documents from DocRED distant\.

Missing inverse relations are added automatically, resulting in the addition of72,60972,609triples\. Similarly to invalid triples, solving asymmetric inconsistencies requires the manual evaluation of12,40812,408triples\. Thus, we remove all entities involved in at least one asymmetric violation from the dataset, resulting in the removal of31,17831,178triples\.

As a result, the*corrected*distant dataset contains1,338,4041,338,404training instance \(−167,234\-167,234than DocRED distant\) and101,870101,870documents \(−3\-3than DocRED distant\)\. The code to correct structural inconsistencies inDocREdatasets is available in the online repository\. The code is compatible with any dataset in DocRED format\. The repository contains the rules specifically developed for DocRED, but the code can also run on user\-defined constraints\.

### 5\.2Experimental Setup

To explore the effect of structural violations on training, we leverage twoDocREmodels with leading performance and open\-source implementations: ATLOP[25](https://arxiv.org/html/2608.20856#bib.bib4)and DREEAM[11](https://arxiv.org/html/2608.20856#bib.bib3)\. We chose two transformer\-based models because graph\-based models cannot scale to the size of distant datasets\. ATLOP exploits a BERT encoder to represent entity pairs enriched with localized context pooling\. During prediction, adaptive thresholding is introduced to better handle multi\-label classification[25](https://arxiv.org/html/2608.20856#bib.bib4)\. DREEAM is a teacher\-student model, whose architecture is based on ATLOP enriched with an evidence\-aware distillation mechanism[11](https://arxiv.org/html/2608.20856#bib.bib3)\. We train both models on the DocRED distant dataset and its*corrected*version, using the ReDocRED development dataset for validation\. DREEAM follows a teacher–student paradigm, where the student model follows a first step of “*self\-training*” and then is fine\-tuned on manual data\. During self\-training, the student model is trained on distant data and weakly supervised signals from the teacher model\. Since our focus is on evaluating the quality of DocRED distant, we consider the DREEAM student model right after self\-training, exploiting the teacher model trained on the ReDocRED training dataset\. We then evaluate each model on the ReDocRED test dataset and exploit their predictions on these documents to assess the number of ill\-formed triples\. The code to analyze the structural consistency of the predictions is available online\. The code is compatible with anyDocREmodel predictions formatted as DocRED official results and can be used with any user\-defined rules\.

Table 2:Structural consistency of DocRE models predictions trained on DocRED Distant \(“*DocRED DS*”\) and its structurally\-consistent version \(“*Corrected*”\) and evaluated onReDocRED test dataset\. For each rule, we report the absolute number of errors in the predictions and its percentage\. Column “*Invalid*” refers to invalid triples, “*Inverse*” to missing inverse triples, and “*Asymmetric*” to asymmetric inconsistencies\.
### 5\.3Structural Consistency in predictions

Table[2](https://arxiv.org/html/2608.20856#S5.T2)reports the structural analysis of predictions by ATLOP and DREEAM models, trained on DocRED distant and its*structurally\-consistent*version, all evaluated on ReDocRED test set\.

Training DocRE models on structurally inconsistent data results in noisy predictions, with ATLOP appearing slightly more sensitive than DREEAM\. The greater robustness of DREEAM may stem from its self\-training procedure, which combines distant data with weak supervision signals provided by a teacher model trained on manual data\. The proportion of invalid triples in the predictions ranges from0\.85%0\.85\\%to1\.61%1\.61\\%, about one third of the rate observed in the training data\. Missing inverse triples in the training set propagate almost linearly to the predictions: while40\.6%40\.6\\%of inverse triples are missing in the dataset, the corresponding figures in the predictions range from41\.18%41\.18\\%to45\.22%45\.22\\%\. A similar pattern holds for asymmetric violations, where the0\.84%0\.84\\%rate observed in the training data closely matches the0\.53%−0\.79%0\.53\\%\-0\.79\\%range found in the predictions\.

In contrast, trainingDocREmodels without logical inconsistencies consistently improves their ability to generate well\-formed triples\. Also in this case, the improvement is more pronounced for ATLOP than for DREEAM\. On average, the proportion of invalid triples is reduced by−62\.92%\-62\.92\\%\. The most substantial improvement is observed in inverse relation prediction: for ATLOP\-RoBERTa, the percentage of missing inverse relations drops by−87\.44%\-87\.44\\%, and for ATLOP\-BERT by−86\.97%\-86\.97\\%\. DREEAM also benefits, with a reduction of−74\.13%\-74\.13\\%\. In both architectures, asymmetric violations are nearly eliminated; Training ATLOP\-RoBERTa on structurally consistent data reduces asymmetric violations from0\.60%0\.60\\%to0\.00%0\.00\\%\.

Table 3:Models trained on DocRED distant \(“DocRED DS”\) and its structurally\-consistent version \(“Corrected”\) and evaluated on theReDocRED test dataset\. Random removal \(“Random”\) was performed ten times with different random seed and the mean performance is reported\. The standard deviation is always below 1%\. The percentage performance improvement compared to DocRED distant is reported between parentheses\.
### 5\.4Effect on model performance

Having established that structural correction reduces ill\-formed predictions, we next examine whether these structural improvements translate into better generalization performance\. Table[3](https://arxiv.org/html/2608.20856#S5.T3)reports the results ofDocREmodels trained on DocRED distant \(row “DocRED DS”\) and its*structurally\-consistent*version \(row “Corrected”\)\.We investigate the impact of each noise source in Appendix[B](https://arxiv.org/html/2608.20856#A2)\.We recall that the corrected dataset contains167,234167,234fewer triples than DocRED distant\. To ensure that any performance differences are not merely due to this reduction in training data, we additionally trained the models on a randomly subsampled variant obtained by removing the same number of triples \(167,234167,234\) from DocRED distant\. This random subsampling was repeated with ten different seeds, and the mean performance is reported in the “Random” row\. To avoid clutter, Table[3](https://arxiv.org/html/2608.20856#S5.T3)does not report the standard deviation, as it is always below1%1\\%\. All models were validated on the ReDocRED development set and evaluated on the ReDocRED test set\.Together with precision, recall, and F1 scores, we also ignored metrics, that measures the precision \(ignPrec\) or F1 \(ignF1\) excluding the entity pairs seen during training\.

Training ATLOP on the corrected distant dataset improves performance across all metrics and both transformer backbones\. Relative to DocRED distant, structural correction yields average gains of\+1\.65%\+1\.65\\%in precision,\+3\.10%\+3\.10\\%in recall, and\+2\.69%\+2\.69\\%in F1\. The improvements are even more pronounced under ignored metrics, where entity pairs seen during training are excluded\. On average,ignPrecincreases by\+3\.83\+3\.83, andignF1by\+4\.59%\+4\.59\\%\.

DREEAM exhibits a similar overall trend, with improvements in all metrics except precision\. Although precision decreases slightly \(by−0\.45%\-0\.45\\%on average\), this drop is compensated by major gains in the other metrics\. Recall improves by\+14\.07%\+14\.07\\%on average, with DREEAM\-BERT achieving a boost of\+19\.32%\+19\.32\\%, while F1 rises by\+10\.02%\+10\.02\\%on average\. Under the ignored setting,ignPrecincreases by\+1\.96%\+1\.96\\%andignF1by\+10\.67%\+10\.67\\%\.

Random removal of triples leads to only marginal improvements in most metrics and even degrades performance in several cases: recall and F1 for ATLOP\-RoBERTa, precision for both DREEAM variants, andignPrecfor DREEAM\-BERT\. These findings indicate that the observed performance gains stem from the correction of structural errors rather than from the mere reduction in the number of training triples or documents\.

## 6Conclusions

We introduced a framework for assessing the structural consistency ofDocREdatasets and analyzed the impact of structural noise on model predictions and downstream performance\. We formalized theDocREtask as the construction of aKGunder theRDFmodel and defined a set of logical constraints inOWLto detect inconsistencies\. In particular, we identified invalid triples through domain and range constraints on object properties; missing inverse relations viaowl:inverseOf; asymmetric violations usingowl:AsymmetricProperty; and relation cardinality violations throughqualified cardinality restrictions\.

Our analysis shows that, compared to the ReDocRED manual training set, the DocRED distant dataset contains three times more invalid triples, nearly five times more missing inverse relations, and twice as many asymmetric inconsistencies\. To better understand the cause of invalid triples, we conducted a qualitative analysis on a representative sample\. Manual annotation revealed that almost70%70\\%of invalid triples stem from named entity misannotations\. This finding suggests that effective denoising strategies should not only target relation labels but also explicitly address entity\-level errors\.

Finally, we examined whether removing structural inconsistencies from the training data reduces ill\-formed predictions and improves model performance\. TrainingDocREmodels on structurally consistent data systematically lowers the number of structural violations in the predictions and leads to improved performance across nearly all metrics, with an average gain of\+6\.36%\+6\.36\\%in F1 and\+7\.62%\+7\.62\\%inignF1\. These results highlight the importance of structural data quality for both the reliability and the generalization ability ofDocREsystems\.

## 7Limitations

To evaluate the performance gains resulting from structural noise removal, we rely on the ReDocRED test set, which itself contains \(a limited number of\) logical inconsistencies\. An ideal evaluation benchmark for isolating the effects of ill\-formed training triples would be entirely free of structural noise\. However, re\-annotating the ReDocRED test set could raise concerns about annotation bias, as such modifications might be perceived as being tailored to favor our approach\.

Our noise mitigation strategy prioritizes entity removal rather than direct error correction\. Correcting all ill\-formed triples would require extensive manual revision of58,61958,619instances \(including invalid triples and asymmetric inconsistencies\), which would be prohibitively expensive in terms of time and resources\. Instead, we chose to discard the affected entities, ensuring structural consistency while avoiding large\-scale manual intervention\. Future work could explore systematic re\-annotation efforts to preserve a greater portion of the data while further improving dataset quality\.

Since we interpret the performance gains from structural correction as the removal of an irreducible noise term in the supervision signal, our error analysis is primarily empirical and does not yet quantify the statistical significance of the observed improvements across different sources of structural violation\. In future work, we plan to complement our empirical findings with a more rigorous treatment, for example by deriving generalization bounds under structurally corrected labels or by analyzing the impact of constraint enforcement on the effective capacity of the hypothesis class\.

Finally, our analysis of training effects is restricted to two transformer\-based DocRE architectures, ATLOP and DREEAM\. Although these models represent strong and widely adopted baselines, future research could extend this investigation to graph\-based and generative approaches\. To support such efforts, we release our code and the full set of structural rules as open\-source resources, enabling the community to reproduce and expand our structural consistency analysis across a broader range of DocRE systems\.

On the sample size for the qualitative error analysis,n𝒮=400n\_\{\\mathcal\{S\}\}=400yields a95%95\\%HPD\(HPD\) interval of±4\.5%\\pm 4\.5\\%around the0\.690\.69entity\-error proportion \(Section[4\.2](https://arxiv.org/html/2608.20856#S4.SS2)\)\. Halving the interval width requiresn𝒮≈1600n\_\{\\mathcal\{S\}\}\\approx 1600, a four\-fold annotation cost without altering the qualitative dominance of entity errors\.

## 8Ethical Considerations

Based on our methodology, we do not anticipate any significant ethical concerns\. We list some potential ethical aspects and risks of our analysis:

- •Licenses and Data Privacy: The dataset and models used in this work are open\-sourced to minimize the risk of privacy leakage and ensure transparency and accessibility\. TheDocREmodels used in the empirical evaluation are trained and evaluated exclusively on MIT licensed datasets, namely DocRED, ReDocRED, UGDRE, and DOREMI\.
- •Hidden Biases: The leadingDocREmodels exploited for empirical analysis \(ATLOP and DREEAM\) are transformer\-based models, relying on pre\-trained language models such as BERT and RoBERTa\. We recognize that pre\-trained language models may include hidden biases from their training data, potentially embedding subtle human prejudices\. While they perform well in detecting relations, special caution is needed for sensitive relation types or entities, where such biases may become more evident\.
- •The use of AI Assistants: ChatGPT, Perplexity, Writefull, and Grammarly were used purely with the language of the paper to help improve clarity\. All scientific content was developed by the authors, and the manuscript has been carefully reviewed and approved in its entirety by all authors\.

## Acknowledgments

This project has received funding from the HEREDITARY Project, as part of the European Union’s Horizon Europe research and innovation programme under grant agreement No GA 101137074\.

## References

- G\. E\. P\. Box and G\. C\. TiaoBayesian Inference in Statistical Analysis\.John Wiley & Sons,New York, NY, USA\.Cited by:[§4\.2](https://arxiv.org/html/2608.20856#S4.SS2.p4.1)\.
- Brownet al\.\(2001\)L\. D\. Brown, T\. T\. Cai, and A\. DasGuptaInterval Estimation for a Binomial Proportion\.Statistical Science16\(2\),pp\. 101–117\.External Links:ISSN 08834237,[Link](http://www.jstor.org/stable/2676784)Cited by:[§4\.2](https://arxiv.org/html/2608.20856#S4.SS2.p2.1)\.
- Cochran \(1977\)W\. G\. CochranSampling Techniques, 3rd Edition\.John Wiley & Sons,New York, NY, USA\.External Links:ISBN 0\-471\-16240\-XCited by:[§4\.2](https://arxiv.org/html/2608.20856#S4.SS2.p3.1),[§4\.2](https://arxiv.org/html/2608.20856#S4.SS2.p4.1)\.
- Dong \(2023\)X\. L\. DongGenerations of knowledge graphs: the crazy ideas and the business impact\.Proceedings of the VLDB Endowment16\(12\),pp\. 4130–4137\.External Links:ISSN 2150\-8097,[Link](https://doi.org/10.14778/3611540.3611636),[Document](https://dx.doi.org/10.14778/3611540.3611636)Cited by:[§1](https://arxiv.org/html/2608.20856#S1.p1.1)\.
- Huanget al\.\(2022\)Q\. Huang, S\. Hao, Y\. Ye, S\. Zhu, Y\. Feng, and D\. ZhaoDoes Recommend\-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),pp\. 6241–6252\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.432),[Link](https://aclanthology.org/2022.acl-long.432/)Cited by:[§1](https://arxiv.org/html/2608.20856#S1.p4.1),[§4\.1](https://arxiv.org/html/2608.20856#S4.SS1.p3.1)\.
- Jiet al\.\(2017\)G\. Ji, K\. Liu, S\. He, and J\. ZhaoDistant supervision for relation extraction with sentence\-level attention and entity descriptions\.Proceedings of the AAAI Conference on Artificial Intelligence31\(1\)\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/10953),[Document](https://dx.doi.org/10.1609/aaai.v31i1.10953)Cited by:[§2](https://arxiv.org/html/2608.20856#S2.p5.1)\.
- Liet al\.\(2016\)J\. Li, Y\. Sun, R\. J\. Johnson, D\. Sciaky, C\. Wei, R\. Leaman, A\. P\. Davis, C\. J\. Mattingly, T\. C\. Wiegers, and Z\. LuBioCreative v cdr task corpus: a resource for chemical disease relation extraction\.Database2016,pp\. baw068\.External Links:ISSN 1758\-0463,[Document](https://dx.doi.org/10.1093/database/baw068),[Link](https://doi.org/10.1093/database/baw068),https://academic\.oup\.com/database/article\-pdf/doi/10\.1093/database/baw068/8224483/baw068\.pdfCited by:[§4](https://arxiv.org/html/2608.20856#S4.p2.1.1)\.
- Lianget al\.\(2023\)T\. Liang, Y\. Liu, X\. Liu, H\. Zhang, G\. Sharma, and M\. GuoDistantly\-supervised long\-tailed relation extraction using constraint graphs\.IEEE Transactions on Knowledge and Data Engineering35\(7\),pp\. 6852–6865\.External Links:[Document](https://dx.doi.org/10.1109/TKDE.2022.3177226)Cited by:[§2](https://arxiv.org/html/2608.20856#S2.p5.1)\.
- Liuet al\.\(2014\)Y\. Liu, K\. Liu, L\. Xu, and J\. ZhaoExploring fine\-grained entity type constraints for distantly supervised relation extraction\.InProceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers,J\. Tsujii and J\. Hajic \(Eds\.\),pp\. 2107–2116\.External Links:[Link](https://aclanthology.org/C14-1199/)Cited by:[§2](https://arxiv.org/html/2608.20856#S2.p5.1)\.
- Luoet al\.\(2022\)L\. Luo, P\. Lai, C\. Wei, C\. N\. Arighi, and Z\. LuBioRED: a rich biomedical relation extraction dataset\.Briefings in Bioinformatics23\(5\),pp\. bbac282\.External Links:ISSN 1477\-4054,[Document](https://dx.doi.org/10.1093/bib/bbac282),[Link](https://doi.org/10.1093/bib/bbac282),https://academic\.oup\.com/bib/article\-pdf/23/5/bbac282/45936115/bbac282\.pdfCited by:[§4](https://arxiv.org/html/2608.20856#S4.p2.1.1)\.
- Maet al\.\(2023\)Y\. Ma, A\. Wang, and N\. OkazakiDREEAM: Guiding Attention with Evidence for Improving Document\-Level Relation Extraction\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 1971–1983\.External Links:[Link](https://doi.org/10.18653/v1/2023.eacl-main.145)Cited by:[§1](https://arxiv.org/html/2608.20856#S1.p8.1),[§5\.2](https://arxiv.org/html/2608.20856#S5.SS2.p1.1)\.
- Marchesin and Silvello \(2025\)S\. Marchesin and G\. SilvelloCredible Intervals for Knowledge Graph Accuracy Estimation\.Proceedings of the ACM on Management of Data \(SIGMOD\)3\(3\)\.External Links:[Link](https://doi.org/10.1145/3725279),[Document](https://dx.doi.org/10.1145/3725279)Cited by:[§4\.2](https://arxiv.org/html/2608.20856#S4.SS2.p4.1)\.
- Martinelliet al\.\(2026\)M\. Martinelli, S\. Marchesin, V\. Bonato, G\. Di Nunzio, N\. Ferro, O\. Irrera, L\. Menotti, F\. Vezzani, and G\. SilvelloA domain\-specific curated benchmark for entity and document\-level relation extraction\.InFindings of the Association for Computational Linguistics: EACL 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 5693–5711\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.301/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.301),ISBN 979\-8\-89176\-386\-9Cited by:[§4](https://arxiv.org/html/2608.20856#S4.p2.1.1)\.
- Menottiet al\.\(2026\)L\. Menotti, S\. Marchesin, and G\. SilvelloDOREMI: Optimizing long tail predictions in document\-level relation extraction\.Knowledge\-Based Systems337,pp\. 115359\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.knosys.2026.115359),ISSN 0950\-7051,[Link](https://www.sciencedirect.com/science/article/pii/S0950705126001024)Cited by:[Appendix A](https://arxiv.org/html/2608.20856#A1.p1.1),[§1](https://arxiv.org/html/2608.20856#S1.p4.1),[§2](https://arxiv.org/html/2608.20856#S2.p2.1),[§4](https://arxiv.org/html/2608.20856#S4.p1.1)\.
- Mintzet al\.\(2009\)M\. Mintz, S\. Bills, R\. Snow, and D\. JurafskyDistant supervision for relation extraction without labeled data\.InProceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP \(ACL\-IJCNLP\),K\. Su, J\. Su, J\. Wiebe, and H\. Li \(Eds\.\),pp\. 1003–1011\.External Links:[Link](https://aclanthology.org/P09-1113/)Cited by:[§1](https://arxiv.org/html/2608.20856#S1.p3.1),[§2](https://arxiv.org/html/2608.20856#S2.p1.1)\.
- Moreyet al\.\(2016\)R\. D\. Morey, R\. Hoekstra, J\. N\. Rouder, M\. D\. Lee, and E\. WagenmakersThe Fallacy of Placing Confidence in Confidence Intervals\.Psychonomic Bulletin & Review23,pp\. 103–123\.External Links:[Link](https://link.springer.com/article/10.3758/s13423-015-0947-8),[Document](https://dx.doi.org/10.3758/s13423-015-0947-8)Cited by:[§4\.2](https://arxiv.org/html/2608.20856#S4.SS2.p4.1)\.
- Ruet al\.\(2021\)D\. Ru, C\. Sun, J\. Feng, L\. Qiu, H\. Zhou, W\. Zhang, Y\. Yu, and L\. LiLearning logic rules for document\-level relation extraction\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),pp\. 1239–1250\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.95/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.95)Cited by:[§2](https://arxiv.org/html/2608.20856#S2.p4.1)\.
- Sunet al\.\(2023\)Q\. Sun, K\. Huang, X\. Yang, P\. Hong, K\. Zhang, and S\. PoriaUncertainty Guided Label Denoising for Document\-level Distant Relation Extraction\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15960–15973\.External Links:[Link](https://doi.org/10.18653/v1/2023.acl-long.889)Cited by:[Appendix A](https://arxiv.org/html/2608.20856#A1.p1.1),[§1](https://arxiv.org/html/2608.20856#S1.p4.1),[§2](https://arxiv.org/html/2608.20856#S2.p2.1),[§4](https://arxiv.org/html/2608.20856#S4.p1.1)\.
- Tanet al\.\(2022a\)Q\. Tan, R\. He, L\. Bing, and H\. T\. NgDocument\-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillation\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 1672–1681\.External Links:[Link](https://doi.org/10.18653/v1/2022.findings-acl.132)Cited by:[§1](https://arxiv.org/html/2608.20856#S1.p3.1)\.
- Tanet al\.\(2022b\)Q\. Tan, L\. Xu, L\. Bing, H\. T\. Ng, and S\. M\. AljuniedRevisiting DocRED \- Addressing the False Negative Problem in Relation Extraction\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),pp\. 8472–8487\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.580),[Link](https://aclanthology.org/2022.emnlp-main.580/)Cited by:[§A\.1](https://arxiv.org/html/2608.20856#A1.SS1.p2.1),[§1](https://arxiv.org/html/2608.20856#S1.p4.1),[§3\.1](https://arxiv.org/html/2608.20856#S3.SS1.p4.1),[§4\.1](https://arxiv.org/html/2608.20856#S4.SS1.p3.1)\.
- Xiaoet al\.\(2020\)C\. Xiao, Y\. Yao, R\. Xie, X\. Han, Z\. Liu, M\. Sun, F\. Lin, and L\. LinDenoising Relation Extraction from Document\-level Distant Supervision\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 3683–3688\.External Links:[Link](https://doi.org/10.18653/v1/2020.emnlp-main.300)Cited by:[§2](https://arxiv.org/html/2608.20856#S2.p2.1)\.
- Yaoet al\.\(2019\)Y\. Yao, D\. Ye, P\. Li, X\. Han, Y\. Lin, Z\. Liu, Z\. Liu, L\. Huang, J\. Zhou, and M\. SunDocRED: A Large\-Scale Document\-Level Relation Extraction Dataset\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 764–777\.External Links:[Link](https://doi.org/10.18653/v1/P19-1074)Cited by:[§1](https://arxiv.org/html/2608.20856#S1.p2.1)\.
- Zhanget al\.\(2024\)F\. Zhang, Q\. Miao, J\. Cheng, H\. Yu, Y\. Yan, X\. Li, and Y\. WuSRF: Enhancing Document\-Level Relation Extraction with a Novel Secondary Reasoning Framework\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),pp\. 15426–15439\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.863/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.863)Cited by:[§2](https://arxiv.org/html/2608.20856#S2.p4.1)\.
- Zhanget al\.\(2025\)F\. Zhang, Y\. Yan, and J\. ChengDocument\-level relation extraction with global relations and entity pair reasoning\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 19556–19567\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1002/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1002),ISBN 979\-8\-89176\-256\-5Cited by:[§2](https://arxiv.org/html/2608.20856#S2.p4.1)\.
- Zhouet al\.\(2021\)W\. Zhou, K\. Huang, T\. Ma, and J\. HuangDocument\-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling\.Proceedings of the AAAI Conference on Artificial Intelligence35\(16\),pp\. 14612–14620\.External Links:[Link](https://doi.org/10.1609/aaai.v35i16.17717)Cited by:[§1](https://arxiv.org/html/2608.20856#S1.p8.1),[§5\.2](https://arxiv.org/html/2608.20856#S5.SS2.p1.1)\.

## Appendix ADenoised Datasets

We perform the same structural consistency analysis presented for DocRED distant on four denoised versions of the distant dataset, generated by two top\-performing denoising techniques\. UGDRE is a label denoising strategy that exploits Monte Carlo dropout to estimate pseudo\-label uncertainty and filter out low\-confidence triples produced by aDocREmodel trained on the manual training data[18](https://arxiv.org/html/2608.20856#bib.bib6)\. Given the manual DocRED training has been revised to produce ReDocRED, UGDRE generates two denoised distant datasets: one exploiting the DocRED manual as the training dataset \(UGDRE\-ReDocRED\) and the other using the ReDocRED training dataset \(UGDRE\-ReDocRED\)\. The second denoising strategy is DOREMI, an active learning system tailored for enhancing long\-tail triples[14](https://arxiv.org/html/2608.20856#bib.bib12)\. Like UGDRE, DOREMI exploits a set ofDocREmodels pre\-trained on a manual training dataset\. Thus, this approach also generates two denoised versions of DocRED distant: DOREMI\-DocRED and DOREMI\-ReDocRED\.

Table 4:Structural consistency analysis of the denoised distant datasets\. For each rule, we report the absolute number of errors in the dataset and its percentage\.### A\.1Consistency Assessment

Table[4](https://arxiv.org/html/2608.20856#A1.T4)reports the structural consistency of the four distant datasets, produced by performing the same syntactic analysis described in Section[4\.1](https://arxiv.org/html/2608.20856#S4.SS1)\.

The percentage of invalid triples is almost doubled in UGDRE\-based datasets than DOREMI\. In particular, UGDRE\-DocRED has almost the same percentage of invalid triples of the DocRED distant \(3\.01%3\.01\\%vs3\.07%3\.07\\%\), hinting at the inability of current denoising techniques to eliminate structural inconsistencies\. About missing inverse relations, the denoised datasets generated by using the DocRED training dataset misses much more inverse triples than the two datasets that uses ReDocRED\. Indeed, the percentage of missing inverse relations in UGDRE\-RedocRED is half of UGDRE\-DocRED \(25\.42%25\.42\\%vs53\.50%53\.50\\%\) while it is three times smaller between DOREMI\-ReDocRED and DOREMI\-DocRED \(25\.42%25\.42\\%vs75\.13%75\.13\\%\)\. This is expected given that ReDocRED solves the false negative issues of DocRED[20](https://arxiv.org/html/2608.20856#bib.bib10)\. Asymmetric inconsistencies are more evident in UGDRE datasets than the DOREMI ones\. For instance, the percentage of asymmetric inconsistencies of UGDRE\-DocRED is double than DOREMI\-DocRED \(0\.77%0\.77\\%vs0\.36%0\.36\\%\)\. Nevertheless, all denoised datasets reports a smaller percentage of asymmetric inconsistencies than DocRED distant \(i\.e\.,0\.84%0\.84\\%\)\. The percentage of cardinality violations are much larger than DocRED distant, which was almost zero \(0\.01%0\.01\\%\)\. DOREMI datasets shows the highest number of errors, peaking at1\.43%1\.43\\%for DOREMI\-ReDocRED\. In general, for both denoising strategies, the DocRED\-based denoised datasets have way less cardinality violations than the ReDocRED\-based\. In particular, UGDRE\-DocRED has almost seventeen times less violations than UGDRE\-ReDocRED \(0\.03%0\.03\\%vs0\.52%0\.52\\%\), while DOREMI\-DocRED has half violations than DOREMI\-ReDocRED \(0\.73%0\.73\\%vs1\.43%1\.43\\%\)\. This behavior may be attributed to the different dataset construction technique with respect to DocRED distant and to the fact that ReDocRED training dataset has much more training triples than the DocRED manual training dataset \(85,93285,932vs38,18038,180\), training the model to predict more positive triples\.

Overall, state\-of\-the\-art denoising techniques fail to mitigate structural inconsistencies and, in the case of cardinality violations, even amplify them\.

Table 5:Structural consistency of DocRE models predictions\. The trained on different denoised distant datasets and evaluated on theReDocRED test dataset\. For space reasons, we use suffix “\-Doc” as a diminutive for DocRED and “\-ReD” as a diminutive for ReDocRED\. For each rule, we report the absolute number of errors in the predictions and its percentage\. Column “*Invalid*” refers to invalid triples, “*Inverse*” to missing inverse triples, and “*Asymmetric*” to asymmetric inconsistencies\.
### A\.2Training effect

This section describes the structural consistency of the predictions of ATLOP and DREEAM trained on the four denoised datasets and their*corrected*versions and their performance\. The correction of the datasets follows the procedure described in Section[5\.1](https://arxiv.org/html/2608.20856#S5.SS1)and the adopted experimental setup is described in Section[5\.2](https://arxiv.org/html/2608.20856#S5.SS2)\.

#### A\.2\.1Models Predictions

Table[5](https://arxiv.org/html/2608.20856#A1.T5)reports the structural consistency of the predictions of ATLOP and DREEAM trained on the four denoised datasets and their*corrected*versions\. As for DocRED, training without structural noise systematically lowers the presence of ill\-formatted triples in the predictions\.

The percentage of missing inverse triples exhibits the most evident improvements, with corrected DOREMI\-DocRED managing to reduce missing inverse relations by−86\.45%\-86\.45\\%on average in bothDocREmodels and configurations\. The improvements in ReDocRED\-based datasets for missing inverse triples are more contained since the percentage of errors in the datasets was already much lower than DocRED\-based datasets, but still ranging from−55\.01%\-55\.01\\%to−83\.22%\-83\.22\\%percentage points\.

The improvements in invalid triples and asymmetric inconsistencies are similar to those reported for DocRED\. The highest improvement for invalid triples is in UGDRE\-DocRED for DREEAM\-RoBERTa, where the percentage of errors drops from1\.37%1\.37\\%to0\.36%0\.36\\%, registering a relative reduction of−73\.72%\-73\.72\\%\. The same configuration and dataset shows the biggest reduction in asymmetric inconsistencies, going from1\.32%1\.32\\%to0\.13%0\.13\\%\(−90\.15%\-90\.15\\%relative decrease\)\. In addition, training ATLOP\-RoBERTa with the corrected version of UGDRE\-DocRED completely eliminates the presence of asymmetric inconsistencies in the predictions\.

Table 6:ATLOP and DREEAM trained on different denoised distant datasets and evaluated on theReDocRED test dataset\. For space reasons, we use suffix “\-Doc” as a diminutive for DocRED and “\-ReD” as a diminutive for ReDocRED\.
#### A\.2\.2Models Performance

Table[6](https://arxiv.org/html/2608.20856#A1.T6)reports the performance of ATLOP and DREEAM trained on the different denoised datasets and their corrected versions evaluated on the ReDocRED test dataset\. While the correction of the DocRED distant supervision dataset produced systematic performance gains across both models and configurations, the denoised datasets did not always benefit from it\. However, such a behavior may be attributed to the evaluation dataset containing some ill\-formed triples \(see Section[4](https://arxiv.org/html/2608.20856#S4)\)\. Overall, there is a performance improvement in at least one metric for all models and configurations, except for DREEAM\-BERT trained on the corrected version of UGDRE\-ReDocRED\. In 47 out of 80 cases \(59%59\\%\), our approach yields performance improvements, demonstrating a net positive effect overall\. Correcting the DocRED\-based denoised datasets exhibit a performance improvement in more cases than ReDocRED\-based datasets\. Considering DocRED\-based datasets, precision and ignored precision \(IgnPrec\) shows an improvement in six cases out of eight – with a peak of\+9\.85%\+9\.85\\%and\+14\.75%\+14\.75\\%respectively for DREEAM\-RoBERTa trained on the corrected UGDRE\-DocRED – recall in seven, while F1 and ignored F1 \(IgnF1\) increase for all models and configurations\.

Table 7:Models trained on DocRED distant \(“DocRED DS”\) and its structurally\-consistent versions and evaluated on theReDocRED dev dataset\. Row “All rules" considers the dataset cleaned from all structural constraints, row “w/o invalid" considers the dataset dataset cleaned from invalid triples only, row “w/o inverse" dataset cleaned from missing inverse triples only, and row “w/o asymmetric" dataset cleaned from asymmetric inconsistencies only\. The percentage performance improvement compared to DocRED distant is reported between parentheses\.

## Appendix BIndividual Structural Constraint Impact

We performed a study on the impact of each noise source to decide which structural inconsistencies to include in the study\. In particular, we trained ATLOP on different versions of the DocRED distant dataset, where we corrected a single structural inconsistency \(invalid triples, missing inverse, or asymmetric inconsistency\)\. Table[7](https://arxiv.org/html/2608.20856#A1.T7)reports the micro\-averaged performance of ATLOP and DREEAM trained on the different versions of the DocRED distant dataset evaluated on the development dataset\. We report the performance of the original distant dataset \(row “DocRED DS"\)\. We also report the performance when cleaning only a single constraints\. For instance, row “w/o invalid" reports the performance of ATLOP and DREEAM trained on the DocRED distant dataset cleaned from invalid triples only\.

Correcting individual sources of noise yields consistent performance gains for ATLOP in both transformer configurations, with the magnitude of improvement varying by noise type\. Invalid triples represent the predominant source of noise; for instance, in ATLOP\-BERT, correcting invalid triples of the DocRED distant dataset results in a precision improvement of\+0\.35%\+0\.35\\%, recall\+0\.87%\+0\.87\\%, and F1\+0\.73%\+0\.73\\%\. The gain is even more pronounced in ignored metrics \(ignPrec\+2\.08%\+2\.08\\%andignF1\+1\.27%\+1\.27\\%\), when entity pairs seen during training are discarded\. Addressing the missing inverse triples results in the highest recall increase \(\+3\.55%\+3\.55\\%in ATLOP\-BERT\), but shows a slight decrease in terms of precision \(−0\.68%\-0\.68\\%in ATLOP\-BERT and−0\.11%\-0\.11\\%in ATLOP\-RoBERTa\)\. Integrating all corrections cumulatively achieves the highest performance gains \(precision\+0\.60%\+0\.60\\%,ignPrec\+2\.11%\+2\.11\\%, recall\+4\.05%\+4\.05\\%, F1\+3\.41%\+3\.41\\%,ignF1\+3\.41%\+3\.41\\%in ATLOP\-BERT\)\. For ATLOP\-RoBERTa, correcting all structural inconsistencies registers the highest average improvement across all metrics\. The configuration shows the best precision improvement \(precision\+2\.66%\+2\.66\\%,ignPrec\+5\.77%\+5\.77\\%\) and the second best gain in recall \(\+2\.85%\+2\.85\\%\), F1 \(\+2\.80%\+2\.80\\%\), andignF1\(\+3\.82%\+3\.82\\%\)\.

Correcting different types of rules for DREEAM shows the same performance trend as in ATLOP; invalid rules are the most influential source of noise, and inverse rules boost the recall\. Applying all structural constraints together achieves the best results, showing a significant performance improvement in recall \(\+15\.52%\+15\.52\\%in DREEAM\-BERT,\+7\.05%\+7\.05\\%for DREEAM\-RoBERTa\) and F1 \(\+10\.85%\+10\.85\\%in DREEAM\-BERT,\+4\.95%\+4\.95\\%for DREEAM\-RoBERTa\), while registering a small precision degradation \(−1\.54%\-1\.54\\%in DREEAM\-BERT,−0\.68%\-0\.68\\%for DREEAM\-RoBERTa\)\.

BothDocREmodels and both configurations demonstrate that each source of noise has a significant impact on the model performance: removing invalid triples improves precision, while adding missing inverse relations increases recall\. However, integrating all structural constraints together harnesses the strength of each individual rule, producing the best overall performance\.

## Appendix CDocRED Structural Constraints

This section reports the structural constraints exploited for our study\. The rules are also available in JSON format in Github\.555Folder “data/rules/” at:[https://github\.com/mntlra/DocRE\-StructuralConsistency](https://github.com/mntlra/DocRE-StructuralConsistency)Tables[8](https://arxiv.org/html/2608.20856#A3.T8)and[9](https://arxiv.org/html/2608.20856#A3.T9)report the domain and range entity type allowed for each DocRED relation\. Table[10](https://arxiv.org/html/2608.20856#A3.T10)reports the relations with an inverse in DocRED\. To conclude, Table[11](https://arxiv.org/html/2608.20856#A3.T11)reports the relations with an upper bound on the maximum cardinality\.

Table 8:Domain and Range constraints of DocRED relations \(I\)\. For each relation, we report the Wikidata ID and a textual description\.Table 9:Domain and Range constraints of DocRED relations \(II\)\. For each relation, we report the Wikidata ID and a textual description \(column “Name”\)\.Table 10:DocRED relations with an inverse relation\. For each relation, we report its Wikidata ID, its textual description \(“Name”\) and the Wikidata ID \(“Inverse ID”\) and description \(“Inverse Name”\) of its inverse relation\. Note that inverse relations also include symmetric relations\.Table 11:DocRED relations with bounded maximum cardinality\. For each relation, we report its Wikidata ID, its textual description \(“Name”\) and the Wikidata ID \(“Inverse ID”\) and description \(“Inverse Name”\) of its inverse relation\. Note that inverse relations also include symmetric relations \(Wikidata ID = Inverse ID\)\.